Hacker News new | past | comments | ask | show | jobs | submit login
Bloomberg’s ‘The Big Hack’ (daringfireball.net)
281 points by okket 5 months ago | hide | past | web | favorite | 173 comments

Regarding Apple's denial, there are other publications that corroborate the Bloomberg story. Previously, Apple has denied security incidents even when multiple outlets report it. For example, last year, The Information reported Apple discovered malware on Super Micro servers in their development and production environments [1]. As a result, the Information claimed that Apple ended up terminating its relationship with Super Micro.

In response to the report, an Apple spokesperson denied there was a security incident, stating: "We’re not aware of any data being transmitted to an unauthorized party nor was any infected firmware found on the servers purchased from this vendor."

However, based on sources from within Apple, Ars Technica claimed Apple employees did find compromised firmware in Apple's design lab. Super Micro SVP of Technology also reported Apple terminated its relationship with them.

I believe we are seeing the same situation here.

[1] https://arstechnica.com/information-technology/2017/02/apple...

Apple's denial references that report.

Which makes it more curious. Last year, Apple PR denied existence of the security incident, but today they admitted it actually happened. That is a contradiction.

No contradiction.

Read it closely: The old statement said that no infected firmware was found on servers purchased from supermicro, meaning freshly purchased servers delivered from supermicro were clean. The ArsTechnica clarifies in an update to the article that the infected driver was installed later.

The new statement says: "That one-time event was determined to be accidental and not a targeted attack against Apple."

A good way to induce misdirection, no?

It does not seem like this story is true. If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg. Sure, a national security letter could force them to stay quiet, or maybe even to lie to the public and say it didn't happen, but it can't make them criticize Bloomberg. Attacking Bloomberg if the story is true is only going to convince Bloomberg to dig deeper. And the story isn't even that bad for Amazon or Apple - it's much worse for US-China relations than it is for either of those companies.

The key technical detail of what these chips are allegedly doing also does not make sense. From the article:

the chips allowed the attackers to create a stealth doorway into any network that included the altered machines

How can you get around a firewall by using a compromised machine that's part of the internal network?

I don't think Bloomberg reporters are just making stuff up. But the technical confusion here makes me suspect that the government officials who leaked this story just didn't understand the details of a real incident that happened, and in the leaking the story got mangled into inaccuracy.

The actual details on the hardware are also sketchy. Based on my reading of the article, this wasn't a chip swap where one chip is replaced by a backdoored version. The article implies that this was an extra chip so either dozens (possibly hundreds) of engineers were in on the operation from the beginning to slip the chip into the design undetected or the chip was mounted without any changes to the board design. The former is much riskier than backdooring the chips and the latter, as far as I know, has not been done before with a nontrivial chip.

Fitting any chip capable of exfiltrating a nontrivial amount of data onto a modern motherboard without going through many rounds of simulation or significantly impacting performance, while also putting it in a place it is capable of intercepting valuable data is practically impossible. Hell, just getting the right power domains wired to the chip is going to be tough enough.

The most plausible theory I have seen so far it that this chip was embedded in the pcb below an empty pad for the backup flash that loads the management system firmware. Three reasons this would work well: 1. The pad looks empty on casual inspection. 2. The data and power traces are already routed to that spot 3. The management system is already set up to load from this flash if it is populated.

That's sounds like the best theory I've heard so far but I'm still skeptical. I've only worked with simple embedded PCB components like resistors or an inductor substrate, but in my experience if there are high speed signals anywhere near the passive, the design has to account for it from the beginning. There's a few clever ways to monkey patch unpackaged silicon into an existing board but AFAIK they all require access to some precise equipment to create a space for the wafer and wire bond it without causing some obvious physical modifications.

Edit: Based on other posts in this topic, it appears that the chip wasn't embedded in the PCB - it was just placed on top of an empty footprint for recovery flash. If that's the case, then this hack is technologically uninteresting.

It looks to be in this position (on an empty pad) in the first two frames of the animated graphic on the article's page, too.

Placing it on a low speed SPI read/write trace to the flash chip containing the BMC firmware is entirely plausible, and in no way would require "dozens (possibly hundreds) of engineers."

EDIT: https://news.ycombinator.com/item?id=18138638

It's entirely plausible as long as no one is inspecting the final product. The hard part isn't connecting the traces, it's doing so without immediately arousing suspicion when a tech open the server and sees a ghetto engineered chip hanging off the PCB at a weird angle.

I hadn't read the detail about the second flash chip. Obviously, if they're just putting compromised flash into a footprint left empty by the manufacturer on purpose for recovery, it's much easier, cleaner, and can be done with a small group.

How does the bogus chip exfiltrate data out of the system, wouldn't the alarms go off at the network level?

Most network level firewalls don't block/inspect outbound packets at this level, and even if they did they could just tunnel (e.g. ssh) through an open port. I think I also read that there was no actual sign of data exfiltration - they were just sending "keep alives" to the C&C hosts.

This was my question also. Typically you do not allow servers Internet access and even where you do it is via an authenticated proxy to an approved set of URLs. If there was traffic to a C&C this would be come fairly obvious especially in a spray and prey type attack like this where it is hard to control where the infected servers go it is surprising that there is no evidence presented of this being detected. Your servers or endpoints contacting unusual internet addresses and especially setting up tunnels is something any monitoring system would be looking for

But the packets would have a destination that's inspectable isn't it?

It depends. These chips were specifically connected to the BMC/IPMI:

>The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

Is your OOB-management VLAN/network firewalled off from outgoing access to the internet? If not, this chip can make requests out to another host, who gives it code to run. This code can then either collect data and send it back out, or do other nefarious things.

The BMC is always on a maintenance vlan for any organization worth its salt. The software stack on those things doesn't deserve to see the light of the day. If you open it up to the internet, you will get pwned, china or not. So there seems to be something missing. It can't just be a compromised BMC.

Edit: Now that I think of it, a compromised BMC can inject payloads into the running OS, depending on the sophistication of the attack and the defenses of the OS. So it does seem possible.

Keep in mind that the BCM already has the ability to piggyback on the production interface if it decides the dedicated interface isn't connected to anything... what's to stop it from doing this in a mailicious scenario and fire off some garbage-looking dns/icmp/whatever?

Unless you're air-gapped you can usually assume that if the machines are syncing to public NTP then there's a way in and out.

Do people really set all their servers to point to someone else's NTP?

How many organizations actually watch what is coming in and out of their DNS?

I work for a company a couple orders of magnitude smaller than Apple/Amazon and we do. If you're in a production environment we're watching every network request, and there's no public access we don't want.

More anecdotes like this can be found at https://news.ycombinator.com/item?id=18138484 .

> And the story isn't even that bad for Amazon or Apple - it's much worse for US-China relations than it is for either of those companies.

If true, it's pretty bad. For one thing customers will be more wary of buying hardware from China. If they move manufacturing elsewhere (or, for Amazon, source hardware elsewhere) they would have higher costs and lower margins, if they don't and a competitor does they lose business.

On top of that, people are already wary of the privacy implications of these companies having all your data, but most are willing to trust major US companies. If it looks like they can't keep it secure against random Chinese companies engaged in industrial espionage, more customers will conclude that operating servers in-house is the better choice.

Silver linong is it’s probably better for both IT people and the economy as a whole that companies generally operate their own servers.

Unless we use Supermicro kit, presumably?

Unless we use made-in-China kit, presumably.

It wouldn't be that surprising to see Supermicro move their production back out of China after this. Though they've been having a hard time of it recently already -- something about getting delisted from the NASDAQ. I wonder if the two incidents are related at all?

It goes beyond that. If this story were in fact true, what we're witnessing is large companies whose entire business is built upon user trust setting fire to themselves. Who'd ever trust AWS again with anything of consequence if it turns out the denial is false?

I don't buy it as written. There's something else here.

What would lead to more of a loss of trust? Revealing to your customers that you hid a massive hack for a few years (even if the government required you to) or insisting that Bloomberg is lying, when you know all the other relevant players will also insist on the same.

> Who'd ever trust AWS again with anything of consequence if it turns out the denial is false?

I agree it makes little sense to attack and deny rather than just stay quiet or keep it to a minimal PR blurb. But I think you're overestimating the amount of blowback they would get if they're lying. Most people just wouldn't care. It might be the decisive factor in a bigger company deciding where to put something new and them going to Oracle Metal or something, but I am really skeptical their bottom line would suffer because of this one thing.

Also, almost everyone who lies thinks they'll get away with it. Otherwise, why lie?

Maybe AWS and Apple are counting on the top-secret nature of all this to enable their lies. Would US intelligence agencies really come out on the record to contradict the denials and confirm the hack if it meant destroying the credibility of a couple of American companies? The Bloomberg article detailed how the government was not willing to sound the same alarms against an American company as they were against Huawei and ZTE for just that reason. I'm guessing that creates enough ambiguity for false denials to be effective.

Yeah I think people don’t understand how big of a deal this would be. Remember Joseph McCarthy? Now picture that, in today’s politics, only with even a shred of real evidence that US companies and the military were infiltrated. That’s an act of war between two nuclear powers. There is literally no telling what kind of chaos could follow.

I’m not saying it’s guranteed that this story is good—-for one, I find the glossed over technical details a little questionable. But it is definitely plausible that this is a false denial, under orders or not.

Regardless, two points are very clear. Remember Stallman who uses that old X60 with a free BIOS? We all called him crazy when he insisted on not having management firmwares and secretive back doors. Now we have a story claiming they were used in an attack by a US adversary (as well as a long chain of confirmed recent security flaws). The second point is that we have to stop trusting China as the sole source for all electronics. It’s a national security crisis that there isn’t a single US facility making many of the components needed for basic computing platform. The federal government needs to immediately and strongly incentive more domestic foundries. And not the trump style “here be factory, jobs good” but actual targeted planning to bring the whole supply chain back, not just the final assembly everyone talks about.

How would one falsify a denial?

The article claims there is an email trail about this. Those communications could be leaked - that would make it pretty obvious this story is true, if it is.

There could also be simply a picture of the spy chip! It's pretty easy to have stronger evidence than this article has.

Both of those are in control of the companies in question.

I think many comments have gotten far too conspiracy theoryish, but I can't think of any plausible answers to "If they were lying, how would we know?"

> How can you get around a firewall by using a compromised machine that's part of the internal network?

Reverse shell depending on how you have the firewall set up. Most firewalls don't block outbound packets, and even if they do you can tunnel over an open port.

> If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg.

If you are a major company, are you going to use AWS if you believe their hardware is pwned?

Almost evreything has DNS resolvers setup. How many places verify their DNS queries are not full of evil?

I find the lack of real technical information quite surprising, not just on bloomberg, but here as well. The previous big thread that you can scroll endlessly has anecdotes and chit-chat but nothing on the actual technical details.

>If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg.

It makes a perfect sense to protect from a potential devastating blow to stock price.

There would be no blow.

China (or the NSA) hacking into something is non-news. Everybody expects that, especially since Snowden. It's their job, and the news would be them failing to do it.

What you're describing is illegal.

I believe it's possible. This story reminded me about Lenovo's drivers automatic installation: https://www.theregister.co.uk/2015/08/12/lenovo_firmware_nas...

And yes, I had the problem with my Lenovo desktop.

That is my question as well, without getting their hands on the machine, how do the attackers start the attacks? And in case of AWS, since everything is so virtualized, does it make sense to install a hardware backdoor if it cant even map to your true target?

It could probably make more sense as a backdoor when the hackers get hold the access to the physical device, it is however quite incredible to me that this hack is designed for a remote network access, and could go unnoticed from infosec within the company if it is truly sending packets outside the firewall...

> That is my question as well, without getting their hands on the machine, how do the attackers start the attacks? And in case of AWS, since everything is so virtualized, does it make sense to install a hardware backdoor if it cant even map to your true target?

Nation states have the manpower for spray-and-pray attacks. They have less need for precise targeting.

> It could probably make more sense as a backdoor when the hackers get hold the access to the physical device, it is however quite incredible to me that this hack is designed for a remote network access, and could go unnoticed from infosec within the company if it is truly sending packets outside the firewall...

How is it incredible? Infosec isn't perfect. Equifax was hacked in part because their IDS system was offline due to an out-of-date certificate [1]. That system would have caught the exfiltration of data if it had been active.

Also, Amazon AWS must have all kinds of crazy traffic flowing through its network due to its customers. My gut feel is that it would be incredibly hard to characterize that traffic in order to proactively detect many kinds of nefarious traffic. The Bloomberg article stated they were easily able to find the traffic from these implants...but only after they knew what to look for.

[1] https://www.bankinfosecurity.com/postmortem-behind-equifax-b...

You can get around that firewall if the firewall is compromised as well.

Turtles all the way down...

It seems that it required internet access to download additional code, if it didn't have that it couldn't do much at all.

For major major reports like this, reputed newspapers are rarely wrong. See WSJ John Crareyou of Theranos. Theranos vehemently denied it, MSM was with Theranos and at least they weren’t outright against them. Then they did more digging etc.

The Washpo digging on Roy Moore, the NYT digging on Clinton Foundation, etc. these are experienced reporters having 15-30 credible sources, evidence etc. every time this happens, everyone denies and slowly little by little the story finds more evidence and facts and it becomes true.

This is why a free press is soooo important

Why would Apple disclose that millions of their products could be hacked and you’ve lost all your privacy. Who would trust them? They’d lose billions, regulations would come etc. it’s in the best interest for every party to deny.

On the other hand, Newsweek was wrong about Satoshi.

My suspicion is that there are many cases where the Chinese government is actually trying to insert backdoors into things, and that in particular Supermicro really has been compromised by the Chinese government, but the technical details of this chip are incorrect. That explains why so many government officials are eager to leak information to Bloomberg, but at the same time the technical details don't really make sense.

It isn't really in Amazon's and Apple's best interest to lie about this. When Gmail got hacked by the Chinese government, Google was pretty honest about it. China has a lot of resources so you can't really expect companies to fend off 100% of attacks on their own; it makes sense for them to acknowledge this publicly and get help from the US government when needed.

The Newsweek that published that article and the Newsweek that built the brand reputation are entirely separate entities.

The current Newsweek is no more formal journalism than Gawker is, which is to say occasionally they'll get big scoops but most of the time they're in the mud trying to get clicks/eyeballs.

Note that Facebook and Apple have confirmed that they had heard about and actually saw, respectively, security issues with compromised software updates from Supermicro - it's only the chip story they're denying. https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

There's a huge diff between "your web management software is buggy" (pretty much every web management software is kinda crappy and significant number probably would dissolve like wet paper under a serious security audit) and "your motherboard has extra chip which grants full hw level access to outside parties". The former is completely routine and happens nearly every time anybody bothers to do a security audit. The latter, if true, is one of the biggest stories of the decade.

And even when Google was hacked by the US government, they're pretty up front about it and about addressing the problems. https://arstechnica.com/information-technology/2013/11/googl...

Remember that this was during a time when the US Government was not run by incompetent hacks who lie all the time. The current political environment does not foster an open discussion as it used to be under the previous administration.

You aren't talking to Trump in incidents like these. Day-to-day interactions with bureaucracy are unlikely to have changed much.

Do you really believe that Trump hasn't infected the Federal bureaucracy? Federal Agencies are headed by political appointees and their assistants. Any reason to pummel the companies which have a (real or imagined) animus will be met with retribution. Just the other day he was falsely ranting about Google censoring Right-wing outlets

Newsweek's credibility is a long, long ways off any of the other outlets being mentioned here.

> When Gmail got hacked by the Chinese government, Google was pretty honest about it.

That hack involved several companies, most of whom were not up front about it.

So counter to your point, Bloomberg reported on a mid-sized company I worked for. The vast majority of the article was sourced from one disgruntled employee, and was completely untrue based on objective facts. An example I could verify visually was the supposition that executive X was hardly in the office. I sat 30 ft from executive X, and he was there every day.

I'm not saying this particular article is incorrect, just that I wouldn't strongly assume either direction without more information.

Was that a highly detailed front page investigative story? Because I’m specifically talking about those. For other cases, they are wrong quite frequently, some of the nonopinion articles on Bloomberg are often a few rungs above college journalism reporting. All elite papers are the same in that regard. The Nytimes just a few weeks ago they had a story that Nikki Haley spent tens of thousands of dollars on curtains and a possible corruption scandal which was easily disproven hours later. Those are usually junior journalists working on it alone, sometimes creating a story rather than reporting on one and fitting all the facts to match a singular narrative. These ones I’m talking about are different. They have a much higher level of checks and due dillegence.

Edit: I absolutely could be wrong and things could be disproven in this, but just in my experience highly detailed articles like this are usually right on the money or close but just like everything in the world, can’t say anything with certainty.

Yes, it was a major story which got a lot of attention and is still mentioned as a source by other journalists.

I think respectable MSM rarely lie about having sources and sources having said what they claim they said. However, sources can be chosen, and sources can lie, so the question is how deep your trust goes. This is why anonymous sources are always very problematic - you can trust that the paper accurately reported that somebody said something, but you have no idea if that somebody was lying or not.

You mean major major reports like Iraq WMDs?[1] Yeah, the press never gets major stories sourced from government sources wrong.

> Why would Apple disclose that millions of their products could be hacked and you’ve lost all your privacy. Who would trust them?

You've got it perfectly inverted. This is exactly why Apple would come clean if they've been hacked by a nation state and take further steps to protect their customers.

Case in point: This happened to Google.[2] Can you imagine how much trust Google would destroy if they vehemently pretended that the hacking never happened? As opposed to going public ASAP with an action plan, which is what they did -- and it improved trust.

These conspiracy theories are never very well thought through. Conspiracies leak. Remember how Watchmen ended?

[1] https://theintercept.com/2015/04/10/twelve-years-later-u-s-m...

[2] https://en.wikipedia.org/wiki/Operation_Aurora

Agreed, I think you've stated your point better than I have further down in the thread based on my down votes.

The American press is not perfect, but over the years I've been really impressed with the journalistic skill of our major publishers. Just this week the New York Times had an extensive story regarding the president's tax returns, they claim to have access to hundreds of thousands of documents and spoke to dozens of sources. I'm more inclined to believe Bloomberg's journalistic integrity than the denials of Apple, Amazon and China.

Didn't Apple and/or Amazon (EDIT: not Amazon) categorically deny being involved in PRISM until Snowden's slides showed that to be false?

As I understand it, there was (and still is) some room for confusion here: (1) PRISM was an internal code name so of course they hadn't heard of it. (2) there was apparently some kind of API or data transfer for sending national security requests to the companies involved and returning the results. A vague PowerPoint slide was misinterpreted by reporters as "direct server access".

This is sort of like claiming that you have root access to Google's servers just because you have a Gmail account.

So the companies could deny that, but there is still a question of exactly what was (is) possible using whatever API they exposed, and what sort of legal review the companies do of the requests they get, and whether this review restricts what the government can do in a meaningful way.

Apple was listed on the PRISM slides but not Amazon. https://en.m.wikipedia.org/wiki/PRISM_(surveillance_program)...

I'd also underline how convenient this news is for the current administration - move production back home. The article is much more damaging to the Chinese hardware suppliers than Apple/Amazon which will likely get over it soon.

>They’d lose billions

This is exactly it. There's no cost to deny, deny, deny until it's impossible, then "oops, after extensive investigations we've found the claims to be true on a limited number of devices...".

But Bloomberg is reporting thousands of affected devices and Apple/Amazon have said they've already done extensive investigations and are totally denying it. That means they both would certainly have to be lying. And there's an enormous cost to that. Very different from being vague or leaving an "out" when the truth comes out.

But in both Apple's and Amazon's denials they claim to have already done extensive investigations.

There's a cost to damaging your credibility.

The public has a very short memory. It’s surprising how fast the credibility is damaged and repaired.

Don’t put too much emphasis on big Corp and their “credibility”.

and there is an even greater cost to 1) not working collegially with the NatSec agencies to maintain secrecy or plausible deniability on critical matters, and 2) destroying instantly your customers' trust in your ability to maintain their data privately, the immediate result of confirming in full the story.

As mentioned above, 1) these denials can be later parsed and opened up as necessary with significantly less reputational damage, and 2) they contain some interesting specifics that can make them strictly true but quite misleading, e.g., "Apple has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server" might more accurately read "someone else found these..."

Scary scenario: Apple altered their Warrant Canary in 2014 [1]. I'm looking to see if any of the other companies did the same.

EDIT: Not sure if it's related, but in 2015 Reddit removed it's canary [2].

[1]: https://www.theregister.co.uk/2014/09/20/apples_warrant_cana...

[2]: https://old.reddit.com/r/announcements/comments/4cqyia/for_y...

On the other hand, NYTimes was wrong about WMD in Iraq, and those articles were sourced from numerous anonymous government officials. Turns out, just as in this case, those government officials had an agenda. In 2003, it was starting a pre-emptive war against Iraq, and in 2018, it seems trade wars with China are popular... perhaps we can connect some dots.

How much of an individual user’s data has been compromised? As far as I understand, as long as you’re not using iCloud, you’re pretty well-protected. Is that inaccurate?

Seems odd that national security officials would outright lie about a computer chip being implanted on hardware. Occam's razor tells me Amazon/Apple are trying not to sour ties with their suppliers.

If you’re looking for a motive, how about this administration’s obsession with getting manufacturing jobs “back” to the US? Seems to me that this along with the current trade war, could be part of a strategy to do just that.

Agree. Bloomberg is taking on the two biggest companies in the history of capitalism and China. And they doubled down on their claim after the initial refutations from A&A. So I’m pretty configent they made double extra sure their facts were correct before they published.

The article doesn’t have to be correct, it just has to be well sourced. If it turns out to be incorrect, then it becomes a story about how or why the sources got it wrong.

On the contrary, when the reports concern China, almost none of these media is trustworthy. I’ve been reading those articles in the past, and checking their original source of information or comparing reports/announcements from multipe sources, only to find that those reports are very often full of information from unreliable sources, misunderstanding or plain lies. Even when they report some truths, they can be very selective on the truths to report to reinforce their existing opinions. Reading those reports about China is very like reading Foxnews. I basically stopped reading those now.

Free media is nice, but articles are written by normal people. It’d be silly to trust whatever you can read

Can National Security Letters can be used to require companies to issue outright lies to the public? The bloomberg article indicates that the investigation is not complete, so that could be on explanation for the apparent disconnect between a seemingly well-reported story and the unsually forceful denials.

Today's Money Stuff ponders another angle of this: whether those companies would be committed securities fraud by abiding by such a request:

> I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here.


What if the story is bigger than Bloomberg or anyone else publicly knows at this point? That is, if the government turned over this rock and found something worse.

The US Government could easily find it in its self-interest to make a deal, involving the SEC, on a national security basis, to allow Amazon and Apple to go the denial route with the US Government giving them guarantees regarding fallout.

If it's worse than Bloomberg has reported, it would be highly desirable by the US Government to keep the rest as quiet as possible for as long as possible, to get at as much of what China is doing as possible. They might be running a counter intel program by now that relies on something China was doing.

What you're talking about is fantasy.

"The US Government could easily find it in its self-interest to make a deal, involving the SEC, on a national security basis, to allow Amazon and Apple to go the denial route with the US Government giving them guarantees regarding fallout."

Can you point out a legal mechanism for this to be possible?

Unofficially, the policy is called "Too Big to Fail".

What's more likely is someone on the inside of one of these companies was privy to the discovery, but not the follow up.

I've been in such a situation myself where I was in the room during what LOOKED like a DDOS by Akamai 10 minutes after we got off the phone with them to turn down their CDN services.

In much the same way as Apple is refuting this claim, after a few weeks of internal debate above our pay grade we decided we didn't see it and it didn't exist and therefore it didn't.

Perhaps these things aren't handled at the level of "company"?

For a national security-related issue, you might just include the minimum number of people that need to know, which would naturally not include your PR team. Then when an article like this comes out, the PR team responds in exactly the way they would if it were an outright fabrication, which is what the goal would be.

Even though this could be the case (that the security team is under a gag order and the marketing/PR team don't know about it), I'm pretty sure a "no comment" response from security would signal the PR team to maybe downplay the whole thing. The active refusal is a bit too strong for just being a miscommunication.

NSL-type deals are explicitly denied. E.g. Apple wrote:

> Finally, in response to questions we have received from other news organizations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.

No but the gov might be unhappy that the secret is out while they were trying to use the chips to infiltrate/reverse engineer wherever the chips were reporting to? Amazon has gov contracts it wants to protect so Im not surprised they would deny this at the govs request, and Apple's #1 product is privacy so having something like this show up on their servers would undermine that product strategy so I can understand their denial too (homegrown or at the govs request)

This seems like the most likely reason for domestic companies to deny everything, and for their supply chain distributors to play along.

1) Big company partners with distributor. 2) Distributor has security issues. 3) Gov is already aware of security issues, says nothing. 4) Big company becomes aware of issues. 5)Gov steps in and pitches a deal: i)Both big company and distributor must deny. ii)In return, gov gets to: iii)Preserve any existing contracts iv)Protect the big company and distributor, with any legal, trade, or commercial benefits

> Amazon has gov contracts it wants to protect so Im not surprised they would deny this at the govs request, and Apple's #1 product is privacy so having something like this show up on their servers would undermine that product strategy so I can understand their denial too

Additionally, Amazon and Apple are two of the still allowed to compete western companies in China that they haven't booted out of their markets, it could be that Amazon and Apple also want to retain the Chinese market and don't want to appear like they are helping while they have the cover of the NSL.

Amazon and Apple are caught in a tough spot and the hardware issue/espionage device is definitely true. It is probably just that the people that know in the company aren't allowed to say and the PR team is not part of that 'need to know' group. Either way, trust of Chinese manufacturing is over, which hits Apple very hard in both market and domestic trust.

I would say not. If the outcome of the investigation is that they were hacked and knew they open themselves up to all sorts of litigation.

What's stopping someone to sue after buying stock under the assumption that the companies are sound? I assume they have a duty to the shareholders.

> But in my experience, Apple PR does not lie. Do they spin the truth in ways that favor the company? Of course. That’s their job. But they don’t lie, because they understand that one of Apple’s key assets is its credibility. They’d say nothing before they’d lie.

"Saying nothing" is indeed an extremely common move for Apple PR. "An Apple spokesperson declined to comment" is such a proverbial reaction that it would be considered completely standard operating practice here.

If the US files injunctions for relief, federal court (FISA) justices can probably compel you to do just about anything. Presumably the Bill of Rights still pertains and takes precedent. But it's not ludicrous to think that you could be compelled to issue a denial or not issue an affirmation of the public claims.

I'm pretty sure FISA has no relation to any of this.

That's what I was wondering too.

Bloomberg reported that the startup I worked for was for sale and the founders were pitching it to potential buyers. Internally, they vehemently denied the report and said Bloomberg completely made it up. Bloomberg was dead on the money. That's not to say every journalist/article they publish is going to be spot on, but I definitely give them enough credit that I will believe this report until proven unequivocally not true by the accused.

edit: spelling.

A CEO shopping a private startup has every incentive to lie to his employees in that scenario and little reason not to.

On the other hand, Apple and Amazon have huge reasons not to blatantly lie about this. The plaintiffs bar would be all over both companies for material false and misleading statements if this turned out to be true (see also Elon's recent experiences with the SEC).

Apple and Amazon are certainly incentivized to not admit bad facts, to spin facts, to issue misleading unclear statements that read as a denial but are not, etc. But I really don't think the legal teams at either company would let executives get away with issuing such full-throated and clear denials if they were untrue.

yeeaaah...on the other hand, they could be pulling a hat trick from the FCC playbook. "I had no idea it was legitimate users leaving feedback and the guy I trusted to inform me no longer works here. I was told everything was fine!" Plausible deniability goes a long way when you have enough money/political support or are "too big to fail". It happens. Often.

I think you might mean "unequivocally"

Thank you. Spell check didn't give it to me and I was too lazy to Google it.

What if Apple or Amazon didn't find the chips but other investigators found them in Apple/Amazon servers. That would make their denial wording technically true. They didn't say no chips were found, they just said they didn't find any.

This is this is similar to all the backdoored Cisco devices the FBI found all over the government. If they are doing it at all they would have a complex plan and this would be one of the many approaches. Even scarier IMHO are the CPU fabrication hacks that add in an imperceptible backdoor directly in the chip logic. A recent report showed how CPUs can be backdoored at critical points in the fabrication process by a single operator that would be incredibly hard to detect. We are talking instruction patterns that charge capacitor buffers that allow privileged access once a threshold is reached. Amazing really.

>But they don’t lie, because they understand that one of Apple’s key assets is its credibility. They’d say nothing before they’d lie.

This is so typical of Gruber as an evangelist.

There's no way that Apple would remain silent on this even if they KNEW it were true. The only possible move is denial.

Silence is validation or uncertainty, a statement of ambiguity will tank the stock and reputation as experts, recognition of even partial truth could possibly destroy their supply lines overnight.

I honestly think the corporate denials here need to be outright ignored because they have so much to lose. A story of this magnitude is basically like pointing a gun to someone's head and asking them for permission to pull the trigger.

I see Apple and Amazon has a tremendous incentive to lie.

What incentive do you see for Bloomberg to report a major story and lie!? Why would they do that?

You can theorize all sorts of conspiracies, like the government planting the story to distance China... so it's possible Bloomberg was mislead.

I think at this point it's more believable the story is true, because Bloomberg is the most credible participant at this point... they're anonymous sources, but as long as Bloomberg did their due diligence as journalists, they've validated their (numerous) sources as credible.

...and like you said, all these corporations have MASSIVE amounts to lose. Just check out what Trend Micro's stock is doing today.

That's exactly my point. They have MASSIVE amounts to lose - that's why they don't want to admit that their datacenters are compromised. Super Micro's stock is down -41%.

> What incentive do you see for Bloomberg to report a major story and lie!? Why would they do that?

just like all of their other stories on Apple products/etc that have been wrong - page clicks and ad revenue. Writing about Apple gets you both. the WSJ is in the same boat. Quoting vague "sources briefed on the matter" is just mysterious enough to keep the reader on the hook for more. I haven't trusted Bloomberg to report anything accurate about Apple for years now.

"But in my experience, Apple PR does not lie. Do they spin the truth in ways that favor the company? Of course. That’s their job. But they don’t lie, because they understand that one of Apple’s key assets is its credibility. They’d say nothing before they’d lie."

As a few people pointed out in the other thread, didn't they pretty explicitly deny they were involved with PRISM?

PRISM was an NSA-internal codename for a really innocuous automation of servicing legal warrants.

Apple knew nothing about anything they were told was called PRISM, and they happily acknowledged the existence of APIs to service warrants.

PRISM was an internal service for saving responses to subpoenas. Subpoenas do actually exist and everyone is happy to admit to them. They even publish reports about it.

As someone on a slack server I use pointed out: a server wouldn't need to phone home, could have a planned failure and request an RMA, even if the system was wiped when it came back, it could have data stored somewhere secretly, and why you may not find anything in an audit.

AFAIK that wouldn't work against Amazon as IIRC hardware doesn't leave their datacenters without being shredded first.

So no hardware warranty repairs?

AFAIK, no.

It was presented as one of many things Amazon does to keep AWS safe at an AWS day I attended or something.

there's a comment on r/sysadmin that's quite chilling, implying that the state of affairs is far worse than what the report describes:


    I did a penetration test and security assessment for a major electronics manufacturer
    whose parts are likely in every smartphone and laptop. I identified almost certain compromise
    by the Chinese government with full access to modify the manufacturing specs using the
    access paths I identified.

    They chose to bury my findings as it would cause a huge stock hit. Sadly, NDA.
    I'm not surprised in the slightest.

Hard to give an anonymous comment any weight without the slightest verification. Since we know that adversaries of freedom use social media as a disinformation vector, the only thing you can do is ignore them or encourage them to find a way to legitimately disclose the information protected under NDA -- perhaps to the press or a legislator who could help make it possible to invalidate NDAs that keep secrets that make us vulnerable.

I agree that panic is ill advised and trust should be given carefully, but if Bloomberg is to be believed, then it happened at least once - and if it did happen once, it's very likely that it happened more than once, because why not? Most organizations don't have resources to find a backdoor like that, nor they have a reason to search for one (had no reason until today I should say) and in this case we only heard about it because of multiple leaks to the press from government officials.

A lot more coming from this news for sure, but kudos to whoever found these, that's solid tech due diligence!

I'm still waiting for technically literate description - as an electrical engineer I don't know what a "signal conditioning coupler" is - searching DigiKey doesn't find anything under that name - it's not a "common" part, at least in my experience. The part looks like an RF filter look-alike of some sort. But that would normally be hooked into a power rail (and ground) - kind of hard to insert signals into a data line unless it was wired strangely

So tell us what the part is pretending to be and how exactly it it was wired (and what it was connected to - is this another Intel ME backdoor?)

Still no statement from supermicro. Nothing on Twitter. Nothing on the website. That's a kind of strange after the stock price droped so much.


While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue. Every major corporation in today’s security climate is constantly responding to threats and evolving their security posture. As part of that effort we are in regular contact with a variety of vendors, industry partners and government agencies sharing information on threats, best practices and new tools. This is standard practice in the industry today. However, we have not been in contact with any government agency regarding the issues you raised.

Furthermore, Supermicro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.

True, but still, I would like to see a statement on the website or on twitter.

Now it is there yes, but it was not when I checked before (8pm Europe).

Yes, now. But it was not on around 8pm Europe time.

supermicro.com is down for me: "This site can’t be reached"

Try www.supermicro.com. With the www it didn't work for me.

It remind me another security risk "Google's new hardware security key was made by a Chinese company"


> not much bigger than a grain of rice, that wasn’t part of the boards’ original design.

'original design' is hard to verify without help from SuperMicro.

> Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community.

... especially for Amazon. Unless Bloomberg claims here that Amazon got parts of this SKU and compared them to newer parts of the SKU and found differences?

Is it legit to revise your design in terms of changes to passives without rev'ing the part and notifying downstream supply chain. Could the grain-of-rice 'microchip' be a different or new resistor/cap? Could it be logic masquerading as a passive?

> In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips.

Like a passive in a blind/buried via?

Anyways, for everyone who claims "it can't be done", "this is implausible" -- you're probably being a little naive. US intelligence agencies do shipment interdiction and adulterate products for this purpose, [1] why couldn't/shouldn't China do the same?

[1] https://www.theguardian.com/books/2014/may/12/glenn-greenwal...

As a side note/question.

I have no idea how things actually work in companies like Apple or Amazon, but would it be "normal" (at their size/scale and given their surely advanced knowledge in technology) to have inspections on the hardware they use (inspections at hardware level of the kind capable of showing these modifications)?

I mean do they routinely do these checks?

I had the same question when I read the article. It sure was lucky that both Apple and Amazon checked the hardware and found the impossible to find chip.

Or, assuming the rest of the article is true, someone somewhere found out about the chip and had a quiet word with a few select customers of Supermicro (but decided to otherwise keep mum about the whole thing). Which would go some way to explain the company denials, because the truth might be a bit embarrassing —or gagged by a three-letter agency.

OK. So what's the minimum remote system management capability needed in a modern data center? The major cloud sysadmin people should figure this out, write a spec, and insist that's all that goes in. If Amazon AWS and Google wrote a spec for this, the manufacturers would fall into line.

Boards are shipping with way too much remote access capability. It's not like you need to look at system busses via the network. You're not going to debug a broken board remotely, you're going to turn it off and replace it. Now that this is an identified problem, it's time to put IPMI and its ilk back in its cage.

Amazon bids for CIA contracts; Elemental bids for Amazon's contracts; Super Micro bids for Elemental's contracts and the PLA "bids" for Super Micro's contracts. Looks like Ouroboros, the global supply chain episode.

A little suspect when they say " giving them access to the most sensitive code even on machines that have crashed or are turned off."

When it is turned off? A magical chip that works without electricity is much more valuable than any data which could be exfiltrated from servers. A chip that works without power changes the world more than the iPhone... in the iPhone Age

The US needs to make electronics supply chain sovereignty the number one priority of the federal defense budget. Why are we fighting useless drone wars when our country is being attacked on a daily basis in the cyber realm?

If this story is true, it will be an escalation clearly and willfully by the PRC from mere state-sanctioned economic espionage to an act of war against the US.

This reaction from Amazon / Apple is reasonable if Bloomberg is lying.

This reaction also seems reasonable if Bloomberg was telling the true.

> The White House requested periodic updates as information came in, the person familiar with the discussions says.

You can't make this up, at least not in Bloomberg, that too in a front page or cover page write up. Truth could be somewhere in between and I am sure we will have more information in the coming weeks.

Certainly not saying this didn't happen, or creating a false dichotomy that a nation-state has to be a friend or foe.. but evaluate this in the context of: https://talkingpointsmemo.com/edblog/white-house-begins-the-... (in case people get the feeling that this story is being pushed, or feel like it's coming out of nowhere suspiciously close to an election)

Stories like this take many, many months of research -- particularly a story as detailed as the Bloomberg. I don't think it's fair to the journalists to simply explain the story away as a government-led conspiracy to mislead voters in the midterm elections.

The US has placed camera in Xerox copy machines that were used in the USSR embassy before, so it's not surprising at all for hardware hacks. w

The difference in this case is definitely scale.

Is this the tiny chip bloomberg is refering to? Maybe they shorted a bunch of aapl and amzn, I'd suggest SEC to investigate the editor who wrote up this crap


Now with the annoucement from the Homeland Security Department of the US, it's very clear, even to a kid, that Bloomberg is lying. The question is should the authors, Jordan Robertson and Michael Riley, get punishment for creating this fake story? They have already created damages. And they may also have some, let's try some conspiracy, motivation in favor of the Trump's gov... since the report is "very conveniently" appeared on the same day of Pence's speech.

For me it's IRRELEVANT in the report is true or not, the very relevant part is that today's hw is complex enough to be a black box for 99% of buyers so it pose a super-serious security risk. Strong article from publicly known sources are a way to shed the light on such ignored enormous security threat.

The sole solution IMVHO is IMPOSE open hardware and free software by law. We simply can't have our society "nervous system" run on black boxes. It doesn't matter if the rouge in charge is China, USA, a specific vendor or someone else. Our banks, our states, our hospitals, ... relay on such connected black boxes. We also need to re-transfer knowledge from big corporate to PUBLIC, well founded, universities to AVOID dangerous evolution paths like actual IT evolution.

You are welcome to write the software for systems you describe and release it for free. Unfortunately, the economic model of the real world is a little more complicated than you seem to realize.

Open source doesn’t mean free. In addition, companies can still offer support, and other companies would still pay for it.

Tell this Linus.

I love open source software and use it every day. I also donate to organizations who contribute to that community. My issue with OP's proposal, rather, was that those who create software and hardware should be forced to release it for free.

A more practical approach would be to at least be forced to provide the ability to run your own software on the hardware you buy, which is also not a thing for the vast majority of hardware produced. At least not for end users anyway.

The problem is, the only ones who care about such a solution are (some of) end users, like us. All the rest, including big money, including governments, seems to prefer black boxes.

This back and forth between Bloomberg and Amazon/Apple reminds me of the recent allegations against Supreme Court nominee Kavanaugh.

In both situations it's difficult to identify who is telling the truth and is probably impossible to know the truth from the outside looking in.

In both cases I've taken a "who has incentive to lie?" view on the allegations and denials, now this does not mean my hunch is correct at all, but it seems way more likely to me that it's in the best interest of Apple and Amazon to deny this story strongly as it makes both parties look bad and the investigation may be ongoing. I don't believe that Bloomberg made this story up but its understandable to question that maybe their sourcing was iffy, based on Bloomberg's history and the level of sourcing that they have cited I'm leaning towards believing the story as accurate.

I personally don't think this is much like the Kavanaugh situation.

For example, Dr. Ford stepped forward publicly, in full knowledge that a massive machine would swing into action to besmirch and defame her, whereas the allegations about espionage were made in confidence. The six(?) individuals making these allegations are risking nothing at this point.

Also, Dr. Ford's allegations reach back into time and are difficult to conclusively prove,* while this is a situation where physical evidence does exist somewhere. It may not be presented to us at the moment, but if the allegations are true, there were physical motherboards that could have been examined to demonstrate the exploits.

What the two have most in common is that we the public are unlikely to be provided with a full and transparent investigation. The FBI's "investigation" into Dr. Ford's allegations did not involve speaking to her or others who could corroborate aspects of her story. Others from Yale approached the FBI and did not get interviewed.

The same is probably true of this story. Even if the allegations are true, for diplomatic or other strategic reasons, the government is unlikely to shine a spotlight on the details.

* Note that "Conclusively prove," is not the same thing as "Obtain enough confidence based on other reports to make a decision about job fitness."

I think similarly to how Dr. Ford knew she would be questioned and pushed into the spotlight when making these allegations public, Bloomberg similarly would've had to calculate the risk that Apple, Amazon, and China would push back on this story vigorously and try to discredit Bloomberg.

I guess my basic premise is that I'm more liable to believe that Bloomberg is operating in good faith in reporting this story. Now that does not mean that their sources may not be credible, but I'm more likely to believe they published this story in good faith rather than Amazon and Apple denying the story in good faith.

I agree that Bloomberg is reporting this in good faith, and that they corroborated as much as they could have given the information they received.

The credibility of their unnamed sources, their motives, and so forth, this is all unclear at this point. It may turn out that they are courageous patriots blowing the whistle on something very, very big. One can imagine a cover-up at the highest level, with these six individuals risking their careers and possibly their lives to reveal the truth.

Or one can imagine that there are some trade negotiations coming up, and a coördinated effort to plant a story so that politicians can take credit for swooping in and enacting regulations around the security of technology manufactured outside of the USA.

Who knows? I don't.

I'm inclined to believe bloomberg as well - however, "who has incentive to lie" seems reasonable on its face, but is a bad heuristic because you will never know the mental state of all the players, and what their motivations are.

I was originally siding with Apple since they flat-out denied that they ever found a planted chip(and didn't waffle with a 'no comment' or something), but re-reading the statement seems like they may be weaseling out with their phrasing "Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server." Bloomberg claimed that Apple found suspicious chips on the motherboard, which may have not been considered a "server" at that point.

In what way does this story make them look bad? They're allegedly the victims of a state sponsored attack. It would only make them look bad if they were co-operating with the Chinese to enable this.

I think it's in the interest of both companies, especially Amazon as AWS servers are critical to this story, to maintain an appearance of having iron clad security systems both to investors and customers.

Didnt all the companies lie about working with the NSA before the Snowden revelations?

No. Snowden's documents show they worked with the FBI, which we already knew and the companies have always admitted to.

This article shows different: "Blanket denials from Microsoft, Google, and Facebook -- and efforts to clear their names -- are the opposite of what AT&T and Verizon did in response to reports saying they opened their systems to the National Security Agency. " [1]

[1] https://www.cnet.com/news/nsa-surveillance-retrospective-at-...

As I said in my earlier post, those denials are accurate according to Snowden's documents. They integrated with the FBI to process court ordered user data requests. They obviously would have no knowledge of a downstream NSA data processing system or its code name.

Your article itself says that the initial reporting was incorrect and links to https://www.cnet.com/news/no-evidence-of-nsas-direct-access-... which correctly explains how PRISM is an NSA system that processes court ordered electronic wiretaps obtained by the FBI.

This is completely untrue, snowden documents showed full cooperation from many companies with the NSA (not FBI) PRISM program.


Snowden's documents show otherwise. Here is the relevant document itself: http://www.washingtonpost.com/wp-srv/special/politics/prism-...

If Apple and Amazon are lying, they're committing securities fraud.


I mean, what would the impact of them committing securities fraud be? Who would go after them and how much would the ensuing lawsuit cost them? The government isn’t gonna go after them if they asked them to keep it quiet.

Probably significantly less than telling their customers they’ve known about and hidden the fact that their systems were compromised for years (even if it was at the behest of the government).

As long as you're a shareholder you have standing. You can sue them right now.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact