Hacker News new | past | comments | ask | show | jobs | submit login
Making sense of the alleged Supermicro motherboard attack (lightbluetouchpaper.org)
707 points by bellinom on Oct 5, 2018 | hide | past | favorite | 313 comments

Ok, that finally makes a bit of sense about "if" this is true, how it might be carried out. And I agree with the author that the simplest action for a chip on the SPI bus would be to hold the MISO line low during power on to suggest to the BMC chip that its QSPI flash isn't programmed (note that QSPI starts up as 'regular' SPI and then switches over[1]). I would guess that the next thing the BMC would do is assume its on the factory floor and hasn't had firmware loaded yet and so it would attempt to boot via TFTP from some server to start the firmware loading process.

FWIW at NetApp the firmware engineers called the BMC the 'BiteMeChip' because they always caused issues when bringing up a new filer motherboard. They were finicky, were often hard to update, and when misbehaving could completely screw up the system.

So I can see this as a vulnerability but really I'd want to stick some probes on that 'mystery' chip and make sure it wasn't just some LC filter or something which is shunting off noise on the data lines.

[1] https://github.com/ChuckM/stm32f469i/tree/master/demos/qspi

Bruce Schneier talked in 2013 already on BMC security linking to a paper [0] from Dan Farmer who did an Internet scan to find that ~230,000 BMCs were exposed to the Internet of which 90% could be compromised [1], yikes.

  [0] https://www.schneier.com/blog/archives/2013/01/the_eavesdroppi.html
  [1] http://fish2.com/ipmi/river.pdf

Agreed on finally makes sense. Depending on what article it made it sound like a chip that sat between CPU and RAM or some external cache that was intercepting some pattern of data and real time buffering then modifying the output - which at CPU/RAM speeds made no sense.

A chip holding and SPI line down - makes way more sense.

One question: how do you hold an existing line low without drawing lots of current, and without cutting that line first?

Two things help you here; SPI usage and CMOS I/O pins.

The SPI bus can be configured with shared master-in-slave-out(MISO)/master-out-slave-in(MOSI) lines since many SPI chips won't even drive the bus if their chip select line is not driven low. Thus the MISO and MOSI pins usually have a fair bit of buffering on them and will often be connected to the bus with a 1K resistor (either externally or built into the chip[1]). On a 5V system this limits the source current to 5mA. Either way these pins are designed to take some abuse.

Current drive is another issue because typically the output driver, if it isn't open drain, will use a p-type drive transistor which has a harder time passing current than an n-type transistor does. As a result the spec iOh (output current when the device is held high) is much lower than iOl (input current when the device pulls the pin low). So one pin to ground will overwhelm the output driver and pull the line effectively to a low state (not as low as it would if nobody was trying to pull it high, but low enough to read as 0). You will see this technique used in 'wired and' type circuits, where output pins are connected in common to a line, and any one going low will pull the bus low. If they are all logic 1 the bus is logic one, if one or more of them are logic 0 the bus reads logic zero.

[1] Yes I know that on "high speed" SPI ports this is not done because of the parasitic inductance in said resistor rounding out the edges of the data pulses with respect to the clock line thus reducing setup and hold time margins for accurate data transmission.

It's not like the MISO line is tied to a power rail. You just need to sink more current than the output of the chip you are trying to override. So the driver is likely in the single digit milliamp range, and in any case, you don't care if you toast the output driver on that chip. In fact, if you do, bonus!

Ok, but i suppose that in the future chip output pads can have circuitry which can detect a forced output (by measuring current). When this happens, the chip could short-circuit power lines, or superimpose a signal on the power-lines to notify the rest of the system.

> When this happens, the chip could short-circuit power lines

No chip design would ever do this. The chip would incinerate itself.

> superimpose a signal on the power-lines to notify the rest of the system

The rest of the system i.e. the BMC already know that something weird is up.

"Lots of current" in this case would only be about 20mA, generating heat that you can easily dissipate from just the surface of a 0201 resistor. I doubt many microprocessors have pins that can drive higher current than that.

Realistically, in order to drive it low you don't have to bring it down to 0V. Most 5V chips will stop registering logic high around 2.5-3.3V for example.

These days 20mA is a "high current" I/O. When you do the math on an 88-pin package, 20mA * N active outputs gets big pretty fast. The max total I/O current can be found somewhere around page 987 of the data sheet... don't stop reading early... :)

(edit: 88 I/O is a modest size microcontroller sort of chip)

I don't understand how this is relevant to holding a single pin low.

it's even easier than the comment above suggests

> the simplest action for a chip on the SPI bus would be to hold the MISO line low during power on

MISO is master-in, slave-out for anyone not familiar with serial peripheral interface jargon. Usually, the slave quad-SPI memory would send the configuration over this line, so pulling it low should dump the real data to ground.

In another thread Walterbell points out:

From 2016, https://arstechnica.com/information-technology/2016/03/repor....

> Apple has begun designing its own servers partly because of suspicions that hardware is being intercepted before it gets delivered to Apple, according to a report yesterday from The Information. "Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter," the report said. "At one point, Apple even assigned people to take photographs of motherboards and annotate the function of each chip, explaining why it was supposed to be there. Building its own servers with motherboards it designed would be the most surefire way for Apple to prevent unauthorized snooping via extra chips."

The industry was aware of intercepted/modified hardware shipments long before this, also giving reason to be suspicious:

From 2013: “NSA reportedly intercepting laptops purchased online to install spy malware”


Also https://www.wired.com/2015/02/nsa-firmware-hacking/

I think the board that is pictured by Bloomberg is just an illustration.

For example, Elemental never used that board. See my tweets here for the actual specs: https://twitter.com/hugelgupf/status/1048160794565861377?s=1...

Elemental boards come with GPUs. The blade pictured has neither GPUs nor PCIe.

I think the attacks are real.

A year ago, Google announced their Titan firmware security chip[1], which would limit these kinds of attacks. I don't believe they designed and built this chip, and surrounding infrastructure, because of purely theoretical attacks.

Besides that, over the last couple years there has also been a lot of work trying to neuter the Intel ME, because of how dangerous it is. Another example is that Google had also done work to replace the EFI/Intel ME with a Linux kernel [2]. This has limited usefulness against the most recent attack, but it is related (since it's still a chip that has more privileges than the primary CPU).

I suspect that there are a very small number of people in these big companies who are aware of these attacks. It's hard for me to guess why the companies involved would deny that these exist.

1. https://www.zdnet.com/article/google-opens-up-on-titan-secur...

2. https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20wit...

First guess: not being allowed to admit it due to national security reasons and it being an ongoing investigation. On the same day several Russians were exposed trying to attack OPCW. They were exposed by Dutch military intelligence. At the press briefing the UK ambassador was there. Same day US indicts several Russian spies. This to show that these are major, international events and that proper disclosure towards investors is probably on the back burner as long as operations are continuing. It must be maddening for those involved not to be allowed to discuss it, or even having to actively lie about it while government officials leak freely in the game of thrones.

I think the NSL angle to the denials is bullshit - the denials themselves give you all you need.

Read the denials carefully. They don't say the attacks haven't happened. They say they haven't found a variety of things.

The apple denial in particular is interesting as it indicates that they have been corresponding with Bloomberg about this issue for a YEAR. Yet despite the magnitude of contact they refer to, they do not describe the nature of the vulnerability being discussed or refer to specific material elements they've refuted. Their statement regarding Bloomberg's most recent version of the facts isn't on material elements of the story.

Having been involved in corporate risk responses I can say I've seen more vehement denials penned with perfect factual accuracy in respect of larger allegations, where the allegations turned out to be true. Almost every statement of defense looks exactly like this, with easy-win half-truth rebuttals to facts peppered around liberally to call the competence of your opposites into question.

Edit: inserted a missing word.

I'm not sure where you're reading the Apple denial you're referring to, but this is *incredibly clear, detailed, and leaves no room to wiggle:

"Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement."


"If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement."


"No one from Apple ever reached out to the FBI about anything like this, and we have never heard from the FBI about an investigation of this kind — much less tried to restrict it."

If you read those as not material, then I'm at a loss as to what exactly would convince you.

Those aren't clear at all. They're clear to you because you don't see the weasel wording.

"Apple has never found [...]"

So what about third parties/reports/partners/contractors? Have they found anything and is Apple aware of those findings? Not disclosed here.

Are the QC processes in place sufficient to lead us to believe that Apple would/should have found this issue? etc. If not, who cares if they haven't found it.

Before you complain about the semantics, Apple specifically uses the phrasing "We are not aware of any" in respect of investigations, yet they don't use that all-encompassing language in reference to attacks.

"malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server"

What is a malicious chip - could the inserted piece be classified as non-malicious or simply not a chip? What is a hardware manipulation - are they saying they've literally never found any deviation between spec and what they've received? Why place 'purposely planted' as a qualifying requirement there?

"If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement."

They don't indicate what was claimed. They reference a volume of correspondence with a variety of claims instead of the claims. We assume, contextually, that they're referring to the specific chip insertion, but that's not borne out by their statement.

"No one from Apple ever reached out to the FBI about anything like this, and we have never heard from the FBI about an investigation of this kind — much less tried to restrict it."

This is fully plausible and means nothing. They restrict their denial of contact with the FBI to outgoing, indicate they haven't received notice of an investigation of some kind, which doesn't mean there's no investigation - collaborating with the FBI specifically on this point wouldn't involve the FBI disclosing the full scope of the investigation to them.

None of these statements actually deal with the issues claimed. They look like they do, but they don't.

Want a real statement? "Upon being notified of the potential issue by Bloomberg this past year, we have analyzed all of the SuperMicro boards referenced to determine if any unauthorized chip insertion occurred and have found none of the alleged inserted chips or any evidence of associated suspicious network activity."

> So what about third parties/reports/partners/contractors? Have they found anything and is Apple aware of those findings? Not disclosed here. Are the QC processes in place sufficient to lead us to believe that Apple would/should have found this issue? etc. If not, who cares if they haven't found it.

You’re making up new assertions from whole cloth - the original story claimed Apple found the chips then alerted the FBI. Apple explicitly and clearly denies every single aspect of the BW/Bloomberg story, and your objection to their denial is that they didn’t deny actions never presented in the article?

That’s called a straw man, and it doesn’t pass muster...

If the chip was found by a contractor of Apple then Bloomberg has indeed asserted something that is factually wrong. But it is not a materially important error, and calling it a "new assertion from whole cloth" is severely overstating the magnitude of the error.

If someone that is talking about bash called linux for "a unix", I don't think most people would support your stance that they made completely baseless statement.

Are the courts not capable of dealing with silly word games?

If they were, you wouldn't spend millions of dollars hiring people like me to play silly word games.

Just for some perspective on how dismissive your comment is imagine the following statement being said in reference to your team: "What do you mean, you had a network breach? Aren't our ops people not capable of dealing with silly 1s and 0s??"

This is scarier & a lot more plausible than the Bloomberg/Supermicro story I think

Claiming ignorance is the best defense.

It almost feels like Apple tried to bait more information from Bloomberg to identify the person who spilled the beans.

If they are under a gag order, they would simply not comment on it. Lying about it is never required and puts them at risk for shareholder lawsuits.

Apple specifically states that they are not under any form of gag/confidentiality order/conditions:

> Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.

Apple cannot do anything bad to China or their manufacturing stops.

Apple would have to move tens of thousands of highly specific CNC machines from China to somewhere else, set them up, and get their line moving again.

This is why I find the idea that Apple phones are "secure" to laughable on its face. China could kill years of Apple revenue if they ever did something truly offensive to the ruling party.

Apple would roll over post haste if China demanded it.

China could cause huge harm to Apple, but the reverse is also true - how would the US government, other corporations, consumers and investors react to such a news?

The ultimate source of most Chinese factory equipment is Europe and US anyway, under extreme political/consumer pressure electronics factories can be setup in a matter of months in US.

> China could cause huge harm to Apple, but the reverse is also true - how would the US government, other corporations, consumers and investors react to such a news?

They would react as usual: lots of whingeing, no action, and roll over for a belly scratch afterward.

> under extreme political/consumer pressure electronics factories can be setup in a matter of months in US.

It takes 9 months to make a baby no matter how many women you get pregnant ...

Many of these equipment manufacturers have entire facilities dedicated to producing equipment solely for Apple. They simply cannot replace that amount of equipment in any timely fashion.

Edit (external reference): http://www.iphonehacks.com/2016/10/next-iphone-probably-wont....

> Koenig writes. “Apple is such a huge buyer of a particular kind of mill (BT30 spindle drill-tap centers) that Fanuc, Brother and DMG Mori each have factories dedicated to building machines exclusively for Apple.”

Not just the factories. Access to the market.

You could move every CNC machine from the manufacturing line and it would not matter.

These sneaky chips were embedded in the PCB. So this is a supplier trust issue.

Plus they probably did not implant every board, So somebody inside FoxConn had to slip in these 'special' pcb's in known Apple mobo orders..

I am assuming that the PCB's are contracted out. It would be interesting to see where the PCB's came from.

One side of me feels real bad for SuperMicro. The have always been there for the small guys that like to build our own servers and for small OEM shops.

Who else are the real server mobo competitors?

Gigabyte and AsrockRack from my view and they are a very distant second and third place.

If SuperMicro goes down we all lose and IBM, HPE and Dell gains.

The original Bloomberg article mentions subcontractor use in this paragraph, but not names:

“As recently as 2016, according to DigiTimes, a news site specializing in supply chain research, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan and one in Shanghai. When such suppliers are choked with big orders, they sometimes parcel out work to subcontractors. In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe. Eventually, that person says, they traced the malicious chips to four subcontracting factories that had been building Supermicro motherboards for at least two years.”

Right, it's worth remembering that Apple deals with lawsuits all the time. All costs and benefits considered, it's probably the best option for them to lie about this for now, and plausible deniability is always a possibility ("at the time of writing, Apple PR were not aware of x, y and z"). The cost of losing access to the gigantic China market and its manufacturing operations would be much too great.

Could they say if they were?

No. But gag orders do not require the recipient to lie about it. Someone who is under a gag order simply doesn't comment one way or the other about it.

FWIW this is the principle behind warrant canaries. A warrant canary is the practice of putting a statement such as "we have not received any NSLs" in a regular report, and then omitting it once you have received an NSL. Because you've conditioned people to expect its presence, its absence then serves as a signal that you likely have received one (though not proof, as there are other reasons you may have removed it, such as being advised by your lawyer that it's a bad idea as the courts may not look favorably on the idea that, no, you weren't actually violating your gag order, you were technically saying nothing, given that you set up the conditions yourself such that saying nothing is in fact saying something).

What happens when you're legally obligated to comment? Like, if you receive an NSL about something, and then later are subpoenaed/compelled to testify before a grand jury or judicial inquest about that same thing?

Neither grand juries nor coroners "outrank" the Executive (unlike Congress, who I would assume can just throw out the NSL to get testimony, since they have similar powers, like throwing out a document's top-secret classified status to get it read into the public record.) But in both situations, you can still be found in contempt of court if you just say "no comment."

I'm pretty sure that this strategy won't hold up in court.

If you have a sign that indicates that a secret event have not happened, the intent of removing the sign is to indicate that the secret event did happen.

The intent is particularly obvious to the originator of the secret event, so you won't be able to argue in the court that it was entirely coincidental.

Warrant canaries have reasonable precedent in the US because of the First Amendment. See New York Times Co. v. United States.

This isn't true and they probably don't work in any other country in the world. They're explicitly illegal in Australia, apparently.

On the other hand, if I actually have a sign indicates that some secret event have not happened, how can I remove that sign if I truely believe I don't wanna keep telling the world that it have not happened?

If I remove the sign just because I no longer want to keep the sign up and someone takes it to mean that the secret event did happen, when it really didn't happen, do they have ground to sue me for fraud or lying?

I agree.

Could you have a canary community that calls companies once/quarter and specifically asks the NSL question? Seems safer for the companies themselves.

The legal system doesn't appreciate cleverness and you get negative points for trying. Either everything would work or nothing will.

If they were, and couldn't say so, it's unlikely they would specifically say they aren't, they would say nothing about it.

And if they were, you think they would admit it?

They would not have blatantly lied about it in an official statement. That could not have passed legal.

You don't need to blatantly lie to author a rebuttal that doesn't actual rebut the claims against you.

You accuse me of selling pink and purple unicorns to gangsters. I reply that "I have no knowledge of any contracts or agreements relating to the sale of unicorns, horses or horse-related animals from my firm, regardless of the colour, breed or condition of the animals. I categorically also deny having any business dealings with any entity which has been charged on racketeering or any other gang related offenses.

The reality being that I 'rent' pink and purple unicorns to Don Corleone for absurd amounts, then don't revendicate when payments stop.

My rebuttal looks sweeping. I put in language to make it look like the scope of my denial is wide ranging and complete. In fact, I intentionally disclaim things you didn't claim to make it look like the moat of propriety surrounding me is vast.

But it isn't. I did what was claimed. And I didn't lie in my rebuttal.

The problem with your arguments throughout this thread is simple: if a rebuttal is clearly engineered to be deceptive, the courts will not regard it as a valid defense in any subsequent lawsuits from shareholders and customers. That's why rebuttals and denials are normally so vague.

Courts have surprisingly little tolerance for companies who think they're being more clever than their customers, their shareholders, or, for that matter, the judge.

I'd love if that were the reality, but it isn't. A judge will scrutinize a party they believe is acting in bad faith (this is a term of art, but I'm using it in the lay sense here), but you won't show that a party is acting in bad faith because they were linguistically precise during a statement of defense.

You'll show they're acting in bad faith if documentary evidence shows they're baddies.

The theory behind our court system is one thing. The reality is another.

     "No one from Apple ever reached out to the FBI about 
     anything like this," Apple writes. "We have never heard 
     from the FBI about an investigation of this kind."
With language like this, it's the very definition of bad faith if they're lying. These are specific assurances that directly contradict the Bloomberg reporters in specific ways, and the companies know that people are likely to rely on them to their detriment.

Aren't they required to do so in the context of a NSL?

No, they are not required to provide materially false information in direct violation of SarbOx, the SEC, and several other federal agencies and statutes - not to mention the various EU laws surrounding such activity.

If they were under an NSL they would simply not comment on this at all. That would be pretty normal for Apple, so people would probably take it in stride.

No. THere's plenty of case law establishing that you cannot be compelled to outright lie.

Most of Apple's "creative image" is one gigantic lie. Why would they suddenly have scruples now?

I wonder if there isn't some level of national security super mega secret scenario situation where the government requires companies to do something and the government indemnifies them against all risks associated with the required action.

How does the USG indemnify these companies for the massive breach of trust if that's indeed the case? You can't pay for trust.

I would be highly surprised if this wasn’t the case.

hasn't it been the case with NSA and carriers all along?


"Congress granted retroactive immunity to people or companies aiding U.S. intelligence agents."

In this case i'd not be surprised if Google/FB/AMZN/Apple/MS/Supermicro were even part of the CIA/NSA counter-sting - feeding of the false info back to Chinese intelligence through the detected 'mistery' chips.

The carriers never lied about it. They were provided immunity from criminal lawsuits, not civil lawsuits.

Are you sure about that? I recall reading, in reference to canary clauses (https://en.wikipedia.org/wiki/Warrant_canary) that the government can and has required lying. If there's an ongoing national security concern, I would expect that they would.

The article you linked is primarily a disproof of your claim.

It cites some hearsay speculation that Warrant Canaries are illegal, and cites several court cases that say Warrant Canaries are protected and compelled lies are illegal: "West Virginia State Board of Education v. Barnette and Wooley v. Maynard rule the Free Speech Clause prohibits compelling someone to speak against one's wishes; this can easily be extended to prevent someone from being compelled to lie."

--no. That's not how this works.

In all cases? possibly not if it is covered by national security.

The press release specifically said that they are not under any sort of order preventing any disclosure.

And while it is indeed possible for the government to make you stay quiet, forcing you to lie is considered compelled speech, which is a fundamental aspect of current thinking on the 1st Amendment, and not something that is likely to change. No, not even when a three-letter agency wants it.

Also nobody at the either the NSA nor Apple cares about the sort of everyday world news you're mentioning. Maybe lay of the Tom Clancy for a while?

not being allowed to admit it due to national security reasons

Came here to say this. When I read Amazon’s rebuttal my gut said “what if these folks had to respond but weren’t allowed to tell the truth?”. If the allegations went unanswered it could be damaging to Amazon and tip the hand of the spooks. If they answered and said they did find the devices the story could run away from them, the internet mob is good at writing articles with “problematic” in the title.

Still I think this was a calculated leak. Why? Probably to put pressure on China in negotiations.

Amazon Web Services is the one branch of AMZN that is actually profitable.

And the make or break for AWS is the assurance that when you're runnign stuff on one Amazon pizza box, your data are safe even from other users on that same box.

You HAVE to get your assurance from Amazon if you're a corporate customer, for due diligence reasons. If that assurance is established to be a lie, every single corporate customer they have would be obliged to migrate to MSFT/RHAT/GOOG or elsewhere. AWS would be dead.

I would say it's a calculated leak that happens to be materially false, meant to pressure CHina and kicj Bezos in the shin.

Bloomberg specifically says some of their sources are senior national security officials from the Obama administration, and the conversations date back to the Obama administration. It doesn't make sense they would have been planning this leak for years in anticipation of Trump winning and a trade war with China.

I think these attacks are theoretically real. I don't think these specific instances of the attack as described by Bloomberg are real.

The only plausible scenario in which none of Bloomberg, Apple, and Amazon are knowingly lying is one in which only a select few employees at Apple/Amazon knew about this and were talking to the FBI, as you suggested. Except this doesn't make sense. The only way in which a select few employees would know about this and nobody else is if these employees are subject to a gag order. But the only way they could be subject to a gag order is if the government came to them in the first place, said "here's a gag order, now that you have it I'll tell you about the attack". But according to Bloomberg, a random spot check by Apple employees found the chip, and IIRC they said a third-party security audit ordered by Amazon found the chip on the Elemental servers. In both cases, this attack would have been immediately reported up to the highest levels at the company prior to even beginning talks with the FBI. There's no scenario in which an employee at Apple or Amazon was made aware of the attack, reached out to the FBI, and received a gag order, prior to notifying their superiors and having the information make it all the way up to the executive level.

Of course, you could claim that Bloomberg was wrong about how the chip was found and right about everything else, but that's not really plausible. Why would they be wrong about that and right about everything else? How would it even be possible for the government to have determined that Apple and Amazon had their hardware compromised without Apple and Amazon's knowing cooperation? The government doesn't have access to these servers, only Apple and Amazon do.

>How would it even be possible for the government to have determined that Apple and Amazon had their hardware compromised without Apple and Amazon's knowing cooperation?

By having a spy in Chinese intelligence who sells it, then doing inspection when hardware goes through customs.

Super Micro is a US company. The finished product didn't go through customs. And any individual components that did would have been going to Super Micro, not to Apple or Amazon or any one of the 30 allegedly affected companies.

I would really hope that this will give all the RISC-V open source effort some traction in the server / DC market. Certainly big corporations like Google, Facebook, Amazon and others could pull it off.

What’s the difference if that open source risc is manufactured in the same Chinese plant?

More likely a fab in Taiwan than China.

You can run your RISC-V cores on an FPGA if you’re really paranoid. Of course, you’d be sacrificing performance.

It's not the RIC-V cores that would be the concern - 'tis the motherboards, NICs, etc...

Yeah, so everything would need to be open and manufactured in the US.

We have the fabs for it. Some are owned by defense contractors, too.

What if the NSA pressures the manufacturer?

every FPGA is far more closed and secretive about its internals than Intel ever has been about its CPUs.

Because an FPGA’s internals is secret sauce...

Xilinx et al. are open enough about how their chips work imo. You are not going to find something like Intel ME on an FPGA.

They don't even let you use anything except their IDE tools, and even then they are tightly controlled, you have absolutely no idea if they contain an intel ME style firmware.

The largest(?) FPGA brand Xilinx is owned by Intel now, so maybe it'll get better.

Altera was acquired by Intel.

Xilinx was not.

It's just my opinion, and so I held off saying, yesterday, but: The immediate economic outfall (co-morbid with institutional panic) would be so severe that this aspect alone would represent a national security issue.

Supermicro stock dipped down to 50 % or so (20->10).

Supermicro has been dealing with some accounting irregularities for the last year or so. This news certainly hasn't helped, but they were already in a shaky position.


That drop happened in the last two days, just after the story was published.


Or that’s a cover story for the dismantling of the spy operation.

I'm not talking about any effect specifically upon Supermicro.

And... it's just a hypotheses, assuming news of such an exploit is indeed being suppressed.

Also interesting that Apple recently started to switch from Qualcomm to Intel modems.

> The modem represents a milestone for Intel in a couple of ways; it is the first chip to be manufactured solely in-house and it is Intel’s first chip to support CDMA and GSM.

> Apple’s original plan for the 2018 iPhones, via Nikkei, was for Intel to have exclusivity on modem orders for the first time — amidst its legal disputes with Qualcomm.

Qualcomm was trying to milk Apple too hard.

> It's hard for me to guess why the companies involved would deny that these exist.

My guess is that they know that everyone's confidence in them would crumble. Every business, ever household, everyone, would suddenly be aware that their data is not safe, even in the hands of the ones who say "trust us, we'll keep it safe".

> My guess is that they know that everyone's confidence in them would crumble.

Why? Even if this were all true, the sum total of the accusation is that a small number of servers 4 years ago were compromised and Apple's security chops are so good they found a completely never-before-seen hardware attack and stopped it with the help of the FBI - all without any customer data being at risk.

I think the attacks are real and if China is doing it then anyone else may be doing it as well including the US.

Anyone else literally can't because they don't have supply chain advantage that China has.

Not anyone else, but there certainly are other nations with similar capabilities.

USA is one of them - China is best suited for inserting "extra hardware" on a board as in this example, but if you'd want to insert similar functionality as some extra stuff in an existing chip, then USA could arguably do it easier than China.

Also, South Korea and Taiwan have immense role in the supply chain and can do similar large scale things as China.

However, yes, the lack of supply chain influence does make similar attacks much more difficult for a bunch of actors which are otherwise active in tech security area - Russia, Israel, North Korea, UK, etc.

> then USA could arguably do it easier than China

USA would be easier to modify servers shipping to US, target specifically at datacenters.

China needs to guess which batch of the contract manufacturing motherboard would go. It would be risky to leak the board tamper to everyone.

NSA intercepted routers to do pretty much the same kind of hack. https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa...

5 eyes/Taiwan/some others could do in on the chip level I guess. Who knows what’s in the me or really all the SOCs...

But can't they take advantage of the vulnerability created by the chips, without caring about who put them there?

but where are said chips being made?

It was reported that the security auditor used during Elemental's acquisition detected this compromise. I assume they found it in a randomly selected board. Either they were very lucky, or hundreds of boards were compromised.

Now, I think all motherboard manufacturers - and especially high end server manufacturers like SM - use sophisticated automated tests and quality control on boards. Under what circumstances is it possible that SM's QC missed this out on so many boards? Won't it affect things like the power budget, weight, and latency? What do professional EE people here think?

    > ... high end server manufacturers like SM - use sophisticated automated tests and quality control on boards.
MFG test guy here (not supermicro!)

Not necessarily. Automated production tests are there to configure and exercise the system and confirm it works as specified. Such tests are good at things like finding bad solder joints, pick-and-place mishaps, misconfiguration of firmware and weeding out product that fails functional test. That, by itself, is hard enough.

Such tests are not _really_ able to detect things which no one is expecting, or worse, things which an adversary has specifically designed to avoid detection. Sure, if something gets "discovered" during a failure analysis, a test or process can be created to specifically address THAT problem in the future.

To find "unknown unknowns", you need an audit of some kind and that is definitely different from production test.

In other words, most automated tests are regression tests.

Thank you for the insight. I was rather naively imagining that every board is subject to some kind of x-ray image matching with the original PCB design to find differences. Not that simple, I see.

Some manufacturers do X-ray boards.

Typically this is done to check that BGA devices (https://en.wikipedia.org/wiki/Ball_grid_array) are soldered properly. Usually not done on every board, but only if there's a problem suspected. When it happens they tend to focus only on particular BGA components or suspect copper traces rather than the whole PCB.

I'm guessing the answer is obviously yes that this type of x-ray would easily be able to discover / distinguish something the size of what's been described? (1mm x 2mm from what I've seen thus far?)

The x-ray picture would contain information sufficient to detect that thing, but it's quite plausible that the process/procedure of analyzing that x-ray would not find it. If they're looking for extra hardware, they're going to detect it, but if they're looking for bad solder joints, they're going to detect bad solder joints but not extra harware.

Right. It doesn't exactly align with what a normal auditor would be looking out for, I suppose. Thanks, I appreciate it.

Usually you scan boards at a percentage determined by statistics (SPC can get complex). However the checks are automated, so you would have to perform the checks in a different trust zone (eg: different country).

I've only seen X-ray as a destructive test. AOI is pretty commonplace on a per-unit base, but whether it could or could not detect some modification depends on lots of details we don't have.

It's not uncommon at all to do AOI on all or a percentage of the PCBs. X-ray is more useful to identify solder issues or defects in the raw PCB itself.

Exactly. I've never seen x-ray used post SMT. We've used it to post-mortemly debug BGA issues.

They would be able to detect rogue component(s) if AOI, Automated Optical Inspection, was done.

However, if the source files already had the component(s) on it, it would pass all tests and QC.

Depends on where it was added. Doing it as part of the assembly process would require having access to the pick and place software along with being able to load reels of the implant into the machine. A well placed insider may be able to make that happen, and automated optical inspections would be fed by the modified pick and place program, so checks would pass as the new component is part of the build.

Thanks for that info. I assumed that once the boards came back to SM's US premises, a second level of optical and other tests are done by SM employees. But perhaps that's not - or was not - their actual workflow.

Would be pretty wasteful to wait until the hardware was shipped to the US to look for manufacturing defects.

EE here, looking at the size of the chip, the weight and power difference is going to be too small to measure. Latency will probably not be affected when the chip is dormant

One of the bloomberg articles claimed that in some cases it was "thin enough that they’d been embedded between the layers of fiberglass" -- like as if it were a passive in a blind/buried via?

This seems like a very sophisticated attack.

Passives don't have enough pins to do much of anything, unless it's an array of passives in a single package that happens to be on power, gnd and data lines at once.

But, I'm not sure but it seems like a bit of a red herring. Reading the article the chip in question doesn't sound passive at all: "The Supermicro board here appears to have a QSPI chip, but also a space for an SPI chip as a manufacturing-time option. The alleged implant is mounted in part of the space where the SPI chip would go."

edit: so maybe it was an array of passives that were meant to go across every pin of this protocol in a single package.

Did you see the comment above that QSPI boots up as SPI and then switches over to QSPI?

It can't be QSPI, he's forgetting the clock and (potentially) chip-select lines.

Not a sophisticated attack, but a standard industry practice for high value, high density boards. I first saw a buried passive 5 years ago.

Sorry, I didn't mean to suggest that the attack was with a passive. Clearly, this package is shown to have logic. But it was meant to look as if it were a passive. In some cases surface mount, but in truly devious ones it's much better hidden.

Well my own idea was along the lines - if they can burry passives, burying a small IC should also be possible

SD cards have the raw IC die buried in the PCB. If you crack open a cheap SanDisk card, you will see that it is a marvel of engineering. Instinctively one thinks of advanced manufacturing as being reserved for high end goods, but in reality it is used widely in cheap mass produced things to bring the cost down.

Are you basing this off the photo in the Bloomberg article that shows a small (0402?) SMT package with three terminals?

I wonder if that's an actual photo of the chip in question.

Based on the captions in the original story, I think it's just a photo of a random signal conditioning coupler, which the chip in question was said to strongly resemble.

But I'm not sure.

FTA: https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iWRg56pmx50...

> [...] a picture of the alleged implant. This shows a 6-pin silicon chip inside a roughly 1mm x 2mm ceramic package – as often used for capacitors and other so-called ‘passive’ components, which are typically overlooked.

It might be possible to mount an attack from a board location occupied by a legitimate passive component (or a single SOT-23 transistor, something like that). In that case the attacker would only need to replace a reel of legit passives in the PnP machine. Visually they'd be identical.

SOT-23 only has three pins, seems like it would be challenging only having one data pin.

The alleged component in question had six pins, if it were passive it must have been a bank or array of passives since individual passives generally only have two pins each.

If it replaced a DNP part, it could have been found at a flying probe test. Half the expected impedance on a few of those pins?

Also AOI might have picked up a difference.

Flying probe testing is unlikely to be used for this type of product.

Sounds like a randomized sample of boards needs tested against a reference then.

AOI would miss the chip if it were buried between layers or look exactly like a part it was replaced for.

I wonder how the auditing team detected it. X-rays?

They might have noted an anomaly in the SMC given the acknowledged SMC firmware exploits (maybe a beta test run?) and investigated the SPI lines to the SMC as a response.

That’s my main takeaway from all of this- Elemental is great at their job and didn’t just do a half assed attempt at an audit. They actually did what they were hired to do. I wonder how rare that is.

You perhaps meant not Elemental, but Elemental's auditors is great at their job.

> ... In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says. Nested on the servers’ motherboards, the testers found a tiny microchip ...

Oh yes, my mistake yes, I thought Elemental was the name of the third party security company. Do we know the name of that company? They need some major kudos for this.

There used to be a Nortel Division up there that had an electro scanning microscope for this sort of thing - I imagine they´re still there under a different name.

> what circumstances is it possible that SM's QC missed this out

Rogue insider, paid or patriotic, likely both.

> affect things like the power budget, weight, and latency

All three would have no measurable change, falling into measurement error margin. Its a death sentence for Supermicro, nothing will help to restore trust. They should publish a detailed post-mortem analysis, though.

They took pains to say that this was happening with sub-contractors in China building motherboards in excess of what Taiwan could handle. So to build trust they can pull back on that extra manufacturing capacity and move it elsewhere maybe South Korea.

> So to build trust they can

You can't put the toothpaste back in the tube

Yeah, SM has the most to lose here. Regardless of whether the compromise is true or non-existent, they really should consider becoming more transparent about their QC, I think.

> They should publish a detailed post-mortem analysis, though.

More of an obituary at this point.

The Bloomberg article says suspicions were raised during a routine audit that resulted in a deeper audit.

The routine audit could have uncovered hackers in Elemental's corporate network doing things like ensure certain machines go to certain customers, etc.

> Let’s assume an implant was added to the motherboard at manufacture time. This needed modification of both the board design, and the robotic component installation process. It intercepts the SPI lines between the flash and the BMC controller

Circuit wise, you don't really need to "intercept" (place in series with) the SPI lines. Two parallel drivers [0] will generally fight it out quietly, and you can guarantee yours will win by designing your implant to have the beefier driver (and perhaps only changing 1's to 0's, as P-channel FETs are generally weaker).

If there already was a footprint for an extra chip (debugging, part flexibility, etc), then it would not require modification of the board design. And an extra component can easily be hand soldered afterwards outside of the standard robotic component placing.

[0] The standard topology of SPI, but in proper operation only one is active due to CS lines.

Post author here. That was what I was thinking of - simply overdriving the lines to drag them high or low in opposition to the 'official' driver. That's pretty much all you can do with 6 pins. I was trying not to overcomplicate the explanation, since the material is already complicated enough!

Fair point about hand soldering, however it would be more obvious to the manufacturing employees for an entire production batch to be diverted to have parts added manually than to simply sneak in another reel to the hundred on the pick and place machine.

I just thought that was phrased as if it were a necessary requirement for the attack, rather than a description of a likely approach. I could also see a scenario where only the occasional unit was bugged, and then an agent at the distributor made sure the right customers received the bugged units.

Thank you for writing your article! I'm currently a bit out of the security headspace and reading the Bloomberg article had me scratching my head like what are the actual details here.

The fact that cursory examination finds the attack is not only entirely feasible, and completely undefended against, but that the hardware shown in the Bloomberg animation is precisely the hardware which would be required to pull off the attack is quite astonishing.

Whose “law” is it that the closer you are to an event the more you can see that the reporting on the event is desperately flawed? This has almost always been my own personal experience.

To have the article hit the nail on the head with a entirely plausible attack means one of two things;

1) This is exactly what happened

2) A nation state wanted us to think this is exactly what happened.

Either way the NSA is involved. The people on the ground who knew it happened would have to be NSL’d from telling their superiors that it happened, so they leaked it.

Perhaps the denials from above are entirely sincere because the people who discovered it can’t pass it up the chain.

The only thing I’m having trouble with is that would have required either a low-level plant to have discovered it and someone else at a low-level found out and was sworn to secrecy, or a middle-manager plant intercepted the message as it went up the line.

The alternative, that this is a false flag, is pretty fascinating in its own right. TAO would have had to conceptualize the attack vector, and then someone planted the story.

Perhaps this is equally likely. In fact, this is actually an attack I assume TAO is using in the field, which would burn this attack vector presumably because they have an even better one.

Even more interesting, the idea TAO was actively using this attack vector, saw evidence in the field their attack was discovered (devices going dark) and so preemptively planted a story to blame the opposition before they accused the US of the same thing.

Or, it could be both.

It could have been a real attack vector that didn't actually infiltrate the companies in question due to having been caught upstream.

But as "attempted attack" sounds far less sexy than "actual attack," the sources may have exaggerated the impact/discovery to further trade negotiation objectives with the spin.

In fact, the BS discovery part of the story might be an attempt to parallel construct a reason it was discovered to cover for a program that might do independent testing of hardware which, if revealed, would give attackers some sense of sampling methodology to be able to counter.

That would explain the vehement denials from the companies allegedly compromised and yet the realistic attack vector published along with the US entity bans on using Chinese hardware around that time.

> Whose “law” is it that the closer you are to an event the more you can see that the reporting on the event is desperately flawed?

idk, but this description of Gell-Mann amnesia seems to touch on that idea


> ...a middle-manager plant intercepted the message as it went up the line...

maybe such a manager would, by doing that in this case, be protecting the top level execs

I'm voting for false flag because it's the most interesting hypothesis to imagine. Especially with the NCSC statement.

A decade ago when I worked at Microsoft I shopped around the idea of using XBox as a basis for secure computing.

XBox was designed to function in the hands of the adversary, to be robust against peripheral attacks and even motherboard mods. Even the main memory was encrypted by the on-CPU controller. Obviously, no open JTAGs. A lot of expertise there.

In my fantasies it would form the basis of the DoD infrastructure and then trickle down to finance. It was deemed to be not solving any practical problems, so it didn't go anywhere. Oh well, I found other fun thins to do in my life.

Are you referring to the original Xbox? Modchips hit the scene for that in ~ 2 years after release. It's not exactly what I would consider a basis for secure computing. Or maybe you are just referencing the procedures "main memory was encrypted by the on-CPU controller. Obviously, no open JTAGs" should be ratified to create a basis for secure computing, a checklist of things to do/prevent before considering a device "Secure".

Didn't even need a modchip. There was a technique where you could softmod by simply unplugging the ide harddrive right after boot and hot plugging it into a desktop.


Ah the old softmod, with the 007 game as I recall. Made a great party trick.

IIRC, the original XBox made the assumption that the main bus was secure due to its high-speed, which turned out to be false. That was one of the few things that was unencrypted, and it offered a way in.

XBox 360 had the on-CPU memory controller with encryption, and it was very tamper-resistant overall. A good model to follow.

I was really impressed with Xbox 360 security. I wanted to use it as a tamper-resistant, PPC desktop. There just wasnt enough demand for underpowered, overpriced desktops secure or not. That happened to folks selling laptops running separation kernels, too. Hardly any people would buy them.

Cool you had the idea, too, though. At this point, Id rather see someone just fund a Freescale implementation of the security parts on their communications processors. Can get more mileage in the market that way.

Anything the government uses that needs to be hardware-secure (like Secret or Top Secret network devices) are either manufactured in small batches in a secure location or it's totally airgapped so that any "additions" don't matter. Let the Chinese have their extra chip installed anywhere they want and you wouldn't get anything out of it anyways, in that situation.

The real problem is for stuff connected to the non-secure internet, like banks, industry, personal info, healthcare, etc.

Would it be technically feasible to have such a small chip which would transfer data out via cellular network instead? Presumably that might not work well in the DC itself, but perhaps for compromising company laptops from supply chain side?

Can someone outline some reasons for me why nobody has come up with an actual physical example of a compromised board? I'm not trying to make a point, I just want to get a more complete picture of the issue, and the biggest thing that stands out to me is the lack of physical evidence.

Well according to the original article, the attack was targeted, implying the only way to get such a thing would be from one of the compromised customers, who are probably involved in an investigation into the matter, don't have all of the info themselves, and aren't super eager to release details to the public before a full picture emerges and mitigation procedures are in place.

Further, going back to the original article, the majority of the information comes from alleged government sources, so not the people directly impacted, but rather those just helping deal with the fallout and coordination.

Assuming there is merit to the story, it will likely be some time before more details emerge, unless having the story out there now helps accelerate that process.

> the attack was targeted,

Supermicro assemble their products in US/Taiwan.

So in theory China need to accurately predict the pattern of how non-China factories install batches of the motherboard. I think it's extremely difficult to pull this off.

Just wait. If they are out there they will be found. Of course you could ask why none of the ANT Catalog software implants have been found yet or why Juniper/Cisco/Fortinet have not (publicly) announced tools to detect them.

It's not like these boards are found in every living room. I would assume that if somebody found a suspicious board at their org, they would kick off internal processes first.

This article is worth skimming/reading in order to fill in the technical gaps in the Bloomberg article; but also describes the complexity of modern computers in a way I didn't previously appreciate.

What's still missing in the overall story, is what Apple and Amazon denials mean. The Bloomberg article says Apple and Amazon independently discovered these chips (Amazon via a 3rd party) in 2015; but the blanket denials by both companies appear to deny this discovery and reporting to U.S. authorities. I'd argue it's unethical for a company to apply falsus in uno, falsus in omnibus to a PR statement, but it's plausible there is a sufficiently misleading or false claim in the Bloomberg article that they feel it's legitimate to dismiss the entire article. In the meantime, the company denials have to be treated as conjecture.

While it's practically certain that hacking attempts do happen from both state- and non-state actors, this particular instance is so "alleged" that it's practically theoretical.

Where's the actual hardware? Why didn't someone decap the tiny chip and probe it? Its design should be well within today's reverse engineering labs' capabilities.

What makes you think that people aren't going through they're POs and server farms and looking today? Just because a couple huge players told prior to publication, doesn't mean everyone was.

These things literally take time, and there's no indication of how widespread the targeting was.

There's a problem with exfiltrate via BMC network theory.

In a sane setup, your BMC connection cannot access internet. You should build an isolated intranet for it (including VLAN or hardware isolation, not just subnet/IP), and put a VPN in the front gate. As a result, you login to your data center, or go to there if you like metaphors. If nobody’s there via VPN, BMC network is a silent and dark place. No connection to outside, no unknown traffic, just silence. Only exception may be the discovery packets of some BMCs, which can find similar servers and form federations for easier management. Even this needs some setup beforehand.

If the BMC has write access to host memory it could surely use that access to create a side channel using the host's network interfaces.

Having said that, it would be nice if networks were segmented in the way you describe. I've been appalled at the lack of segmentation I've seen in companies of all sizes that I've had gigs for.

Today's network cards are small computers of their own. So it'd be very hard to inject packages with all this kernel-hardware integration at the module level IMHO. The card would probably throw a tantrum if you try to access it directly.

TBH, while I'm knowledgeable about hardware, I'm a total beginner in attack side of cybersecurity.

At least, we are segmenting our networks like that.

Blame management.

According to this comment, the BMC is at least capable of working off the real NIC instead, and will do so if you don't hook up the management NIC. https://news.ycombinator.com/item?id=18138411

This is a BIOS setting, and even while the BMC's working over the primary NIC, it retains its independent MAC and IP address (and VLAN if you set it up). Also, even if the NIC is shared with the BMC and the OS, you cannot see the NIC of the BMC on the PCI bus. They are isolated at the hardware level.

I manage lots of these servers for a long time, and this is my firsthand experience. :)

The BMC using the correct MAC and IP address is entirely down to the honesty of the software running on the BMC.

Yes, but if you isolate at cable level, using primary NICs MAC and IP is moot. If you're sharing the medium, you need a good security team.

Attacks involving writing to / reading from memory & exfiltrating that data needs proof of concept code now. Hope someone with comparable hardware and BMC modifies OpenBMC's code to steal some stuff from the OS for research purposes.

If it can be proved, the results will be relatively fun and certainly revolutionary.

Where did all the boards in question go? Why wouldn’t a company notice any of the outbound traffic using firewalls?

Two pieces of the story that don’t add up for me.

> Why wouldn’t a company notice any of the outbound traffic using firewalls?

This is telling you that your experience is limited, not that the story is wrong. Trying to do egress filtering at scale is extremely hard for all but the most basic threats. If they open a socket to data-collector.pla.cn, yes, probably a majority of large shops would notice that within a few months but what if it's just a connection to S3/EC2, buried in the noise of all of the legitimate use of those services?

Think about how hard it is if the attacker is smart enough to bundle that into other traffic: compromise your mail server and have it respond slightly differently to some spambots, have a webserver respond to the Baidu crawler with actual data encoded in the cookies it sets knowing that the Great Firewall can pick the data up (Baidu doesn't even have to be involved – just something which gets packets somewhere they can see them), etc. If it's on a network with people, that's especially easy – is that user-agent hitting sites in China a bot or just the staffer who keeps tabs on market news for that region? (But, you may say, our user and server networks are tightly segregated! Does that apply perfectly to your terminal servers? How would you know if your web proxy started making a few extra requests?)

The mention of seeing a problem first in a Siri data center was interesting to me because those data centers probably have very consistent network activity and it'd seem like the kind of place where the defense team would have the best chance.

General practice for things like BMC and other out of band control systems is to put them on isolated vlans and default deny any non-approved traffic.

There is no way that any large tech company security operation misses this traffic phoning home from a management vlan.

BMCs can often also communicate on the other non-dedicated ports. It is not impossible for a compromised BMC to see traffic on an ethernet port or change its VLAN and go out on the same VLAN as the rest of the normal traffic.

That's true but right now I can think of at least three options where that wouldn't be relevant:

1. The implant can use the host network interface before the host OS starts 2. The implementation depends on VLAN tagging and the implant simply uses configures its network interface to use the same tag as the host interface 3. The implant can compromise the host OS when it loads and use its networking stack after it loads

> Why wouldn’t a company notice any of the outbound traffic using firewalls?

IIRC the original story mentioned en passant that at least one company detected some odd behaviour on the network, which eventually resulted in hw examination and the uncovering of these moles. TBH, that part is immaterial: it might well be some parallel construction to avoid giving away NSA intel capabilities.

> Where did all the boards in question go?

Warehouse 51 [1] of course.


Really, you think it’s easy to spot a few stealthy packets among trillions? You have far too much faith in detection capabilities. Many of the best companies go years without detecting rogue traffic.

Also, just because you don’t know where the boards are does not mean that they don’t exist.

> notice any of the outbound traffic

Outbound traffic is allowed unrestricted in 95% of the deployments, it is just a life fact, people trust their own systems.

Yep, lack of egress filtering everywhere. Scary how many cloud deployments I see that are like this. Improvements in automation tools and the ability of cheap cloud resources enable people to roll out instance after instance with the same basic configuration mistakes.

Yum update, pip install whatever :)

Rubygems for everybody! Or arbitrary docker containers.

My current deployments allow outbound SMTP only. All software packages (rpm or whatever) get pushed in via rsync from the outside, or are built in an adjacent lab behind the firewalls and pushed across.

It's sometimes surprising who gets unhappy when you tell them they can't just run arbitrary code from places unknown in prod.

It may affect the very source box you are using for package downloads. I bet it is allowed to go outside unrestricted. Funny you mentioned Docker, it is a security nightmare in itself.

Indeed, although it is connected via rather restrictive corporate proxy. I'd rather have one box on my network exposed where I can monitor it than a thousand instances with unrestricted outbound access in AWS though.

Oh yeah - docker. I thought this one was pretty funny:


> Why wouldn’t a company notice any of the outbound traffic using firewalls?

If you deploy several thousand of those backdoor chips, you wouldn't set them up to ping your C&C infrastructure all on its own. Rather, you'd use a magic string as a trigger to activate the backdoor in some few targets to reduce the chance of the outbound traffic being detected.

> Why wouldn’t a company notice any of the outbound traffic using firewalls?

The attacker could use this to escape AWS/shared computing sandboxes/containers in order to attack their peers. Exfiltrating the data stolen could be easily hidden in something that looks like legitimate customer traffic.

Huh? The traffic has to travel over wire as TCP/IP, regardless of what device generated it. Any outbound firewall would detect that, especially traffic going to ports that aren’t even open/used.

Many/most AWS customers use ssh sessions to interact with their allocated nodes. And when it's not ssh traffic, it's often https. What good does it do to detect the bad traffic if you can't distinguish it from legitimate customer traffic?

> has to travel over wire as TCP/IP,

BTW, this is not the case. If exfil via conventional system networking is too hard to avoid detection, they'll find another channel. RF via LOS, ultrasonic, or some of a million other ideas.

If I were an attacker in that situation I'd probably find a way to get my data to hitch a ride with other outgoing data

There wouldn't be any weird outbound traffic; the attacker would use another cloud instance as a controller. They would use a US-based account (and IPs via VPNs) to control the controller.

Cloud instances are generally firewalled off from each other, usually cross-account, vpc, etc. Two machines can't reach each other just because they're AWS instances. That would be silly.

The attacker and target have VMs on the same hardware. The attacker has a fake presence serving cat pics which is constantly sending “valid” logging traffic to an S3 bucket.

The chip passes the stolen data between the VM instances by DMA. The stolen data hitches a ride inside an otherwise innocuous TCP packet storing log entries in that S3 bucket.

Logs are routine backed up off site.

There would be absolute nothing at the network level to distinguish the infiltrated packets.

Lower latency scenarios could be devised which would be similarly invisible but allow better command & control.

It seems that you're arguing for a different functionality of the implanted chip than the original article.

Original article supposes that the implanted chip was used to modify the boot process of the BMC in such a way that the BMC loaded its software over the network. The question is why this network activity wasn't easily detectable and firewalled off by default.

It seems that you suppose that the chip can be directly used for some sort of userspace-inside-a-VM to ring0(?) escalation. In particular, you suppose that the chip has access to main memory. Can you elaborate on what functionality do you think the implant could've had?

Not the parent poster, but the chip can indirectly used for such an escalation - the chip would have the capacity to modify BMC software as it's loaded from a nearby memory chip; and BMC has direct access to main memory that enables all kinds of interesting escalation scenarios.

The traffic has to travel over wire as TCP/IP, regardless of what device generated it. It has to travel over wire somewhere! The device can’t magically communicate with China. Any outbound firewall could detect that, especially traffic going to ports that aren’t even open/used.

i gave you the benefit of the doubt when i read your earlier comment, but this doubledown shows you're extremely naive / have no experience with the space. you realize there are entire industries (plural) premised off the fact that you can't just throw up "any firewall" and detect this kind of thing, no?

can you imagine how it would go down if a major corporation like apple experienced a hardware hacking based infrastructure breach, contracted a cybersecurity company, and the tech heading their case asked them like "well were you making sure to check which ports were open?"

I think the idea is that the traffic goes over intranet to another computer that they legitimately control within the datacenter, then goes to China.

Or if they can insert themselves into the maintenance team, they can retrieve their chips at a later date.

Unless, just to provide a nightmare scenario, the routers that would detect it are also compromised.

The NSA did that, they literally intercepted routers en route. I'm guessing other agencies have too.

Or you can just use the default credentials Cisco is addicted to leaving in their code.

Aren't we talking about virtual machines running on this hardware?

Wouldn't be it possible to somehow hide a signature and instructions in legit MPEG payload? Think of video scaling service getting prepared video file with a binary pattern which once detected by hacked HW/FW would activate the hidden instructions and plant results in a response.

I don’t know what you mean by the first question. The implication was that this was a bit of a drift net attack. Compromise a few lots, then wake them up a few months later figure out where they are. Most aren’t useful, but if you got the right batch, some might end up someplace interesting.

I would assume the reason why no one noticed outbound traffic is because it’s dormant.

So if there’s a bunch of compromised boards, where are they? Which boards? If they truly are in a bunch of datacenters, and we know the manufacture and models, why hasn’t anyone just gone and looked? Or Why haven’t the leakers just provided the actual chip? It should have been as easy as ordering one.

Why are you assuming people haven't looked?

> Why haven’t the leakers just provided the actual chip? It should have been as easy as ordering one.

Sure, let me just order up a state intelligence compromise. It's right here on Alibaba right under Official_PLA_61398.

Few years ago https://hackaday.com/2013/08/02/sprite_tm-ohm2013-talk-hacki... popped up, basically hdd have many arm co-controllers. Nowadays everything is a processor, everything is thus virtual and fakeable. Enjoy the paranoia :)

I'm no expert, but wouldn't be more efficient to completely replace the BMC with a malicious one? That way it should be virtually impossible to detect

The cons to that approach is the work it takes to design a BMC without knowing how the existing one is designed. Replicating all of the BMC behavior is hard. Differences in behavior might be easily spotted, whereas this can lie dormant until triggered. "Why does unit #4322 have a faster/slower ping response?" Oops, discovered!

You don't have to redesign the BMC. Just repackage them with additional dice to take over I/O pins as needed.

Or just reflash the firmware. They should, but most people don't reflash the bios and BMC firmware when they install a new server. In fact I routinely encounter servers that are 5+ years in service that have never been reflashed.

Apple, at least, claims they do[1]: "As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections."

[1] https://www.apple.com/au/newsroom/2018/10/what-businessweek-...

They surely don't write their own firmware for Supermicro boards? So "update firmware .. with the latest protections" means update from the hardware vendor?

I know more than one vendor who's dumped firmware from BMCs to compare to the vendor image. Simply reflashing the firmware would be caught by this, an implant wouldn't.

There's quite a lot of embedded firmware that isn't derived or controlled or affected by BIOS/UEFI firmware updates. e.g. Thunderbolt (PCIe) firmware.

otoh you have to consider the probability that flashing the firmware will brick your server (p >> 0.0 in my experience).

Absolutely, and I think that is why a lot of people are reluctant to do it. It's a fragile process and if you lose power or have a random cosmic ray hit at the wrong time it's game over.

In true HN fashion, I’m not an expert either, but I can think of two plausible reasons.

1) Directly compromising via software the BMC is easy to detect if you bothered to look. (i.e. checksums don’t match), and would only last as long as the BMC isn’t flashed.

2) It’s simply currently easier to make a second component, rather than to integrate it into the main chip directly. Of course, if I was in charge of this project, I’d already be investing resources into exploring this option.

3) the purpose of the operation was to produce headlines and fear in the western press. A few bytes changed in some firmware, or even a malicious BMC design (presumably they're gate-array designs, not full-custom) wouldn't allow for macro photos of a chip the size of a grain of rice and all that.

Or even why two SPI roms?

Why not just one with special code? It's not like anyone is routinely reverse engineering the BMC boot code.

It seems like an awful lot of provable trouble to go through (note that there is no physical evidence in the public eye yet) when you could do the same thing, at the factory, with just software.

it's pretty easy to dump an SPI chip and some vendors/customers routinely do so.

In this model, the implant basically contains a binary patch that is injected at boot time over a segment of the BMC binary - counter overflows, chip cuts out the SPI and transmits its payload instead. After the payload is injected the implant goes dormant again/resumes passing through the SPI, so dumping the ROM after boot, or even physically clipping onto the SPI chip will not reveal the wu-tang secret.


>Or even why two SPI roms?

Because AST2400 has input for two flashes, one main, one recovery. And the configuration that uses that is not common. That specific moment actually makes it harder to detect, while being very simple from technical side.

Technical possibility is one thing; proving the story has actually happened is another thing. Until now, what we get is a categorical denial of the story from all related parties. And all the evidence Bloomberg can provide so far is just vague anonymous sources.

Talk is cheap, show me the code/server/chip if they ever exist. Otherwise, the story is just a blunt lie fabricated by Bloomberg serving as a propaganda to bash China amid the Sino-America trade war.

I really hope that this is the straw that breaks the back of all these "management engines"

Like seriously, why does my hobby consumer motherboard need that feature? Corp IT only ever deploys to large fleets of OEM machines.

The management engine in question is an ASpeed standalone part which is only used in large-scale server deployments, not the CPU MEs that you're forced into.

Take a look at ASpeed (BMC supplier) stock movement: https://finance.yahoo.com/quote/5274.TWO/chart?p=5274.TWO

I know, but seeing the article mention "management engines" made me hope that it will direct a bunch of newfound scrutiny on Intel ME/AMD TrustZone/etc

That movement seems to have started about two weeks ago. I don't think it's related to this news.

I'm sure it didn't help!

BMCs have always been an Achilles heel.

HP's iLo could be got into if you used `curl -H "Connection: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA"`

Supermicro's horrid BMCs, of which there are many, all were horrendously lax in security, long before this chip was, or was not inserted. You didn't buy supermicro if you were worried about security, you bought them if you needed to stack stuff high, cheap and dense.

There is a reason why its best practice to put them on separate networks, with as much stuff between it and anyone else. BMCs are massive backdoors, and should be treated with caution.

Does this mean I can get some great deals on Supermicro boards now? :)

Regardless this attack is real or not, this exposure and assessments (just like Charlie Miller and Chris Valasek did for exposing the car hacks) raised good awareness of how vulnerable BMCs are and hopefully the baseboard hardware security and firmware security will be securer in the future when they evolve.

The Register's article on the matter seems also good: https://www.theregister.co.uk/2018/10/04/supermicro_bloomber...

I don't understand why everybody assumes it must be pulling data over the network autonomously. It could simply compromise the host OS to augment other, targeted attacks. Say, by reintroducing a buffer overflow or race condition. It would be incredibly short-sighted to go to such lengths to compromise these machines just to naively pull from a command+control server, virtually advertising its presence.

Defenders simply do not think like attackers. If you're a defender it's lethal to try to think like an attacker. "I can't imagine how this would be useful therefore it must not be useful" are the famous last words of everybody who has cast doubt on a new and novel attack vector. As a defender you need to first estimate the unknowns and unknown unknowns, which over the last year alone have exploded (e.g. actual and potential Intel microcode vulnerabilities). If you needed this article to convince you of feasibility or even just practicality, please don't pretend to be capable of assessing the security posture of any complex system.

And let's be clear: this is not a new and novel attack vector. There are companies that have existed for quite some time researching and selling products to deal with this sort of attack vector. On the spectrum of hardware based attacks feasible today, this chip isn't at the complex end of the spectrum but rather the simple end of the spectrum. The complex end of the spectrum involves hiding logic deep within existing ICs, and there's ample literature to demonstrate feasibility of both implementation and detection.

The difficulty in pulling off these attacks lies not in the software or hardware, but the political, intelligence, and military apparatus of attacking countries. The economic costs of detection are huge precisely because, at the end of the day, fundamental security relies on trust, not technological hurdles per se.

In the security world, there is always a new attack vector. With the advent of Spectre and Meltdown, I am expecting to see more hardware-based attacks in the future.

and subsidies to enrol more students in EE to build up local national talent?

Regardless if someone of the details turn out to be true, there's a couple of really important network security takeaways here.

First you need to run an air gapped network or at least switch enforced vnet for your bmcs. Three firmware on these is only updated every few years anyway, so they're likely full of security holes anyway.

Second, in general, outbound connections need to be monitored everywhere in your data center.

Is this really that hard to imagine?

I am willing to bet that there are teams of spies who have infiltrated Google, Facebook, Amazon, etc. Spies from US, Russia, China, Britain, Israel, Germany, etc, must have dozens of spies working as engineers are getting access to all that data as we speak.

To think that they aren't would be rather naive, in my opinion. If I were head of the spy agencies in any one of those countries, it would definitely be the first thing I would do.

It's as hard to imagine as the NSA's surveillance systems.

I.e., it's not.

The lesson of the Snowden leaks is clearly this: if it can be imagined, and it can be useful, and they have the budget, and it's remotely doable, then it's been done.

China almost certainly did this because they could. You only get one chance to do something like this, so you have to do it even if it risks losing the ability to do it in the future: if you never do it, then you never get the benefit, and if you do do it, you get to do it once, and once is better than never.

Thank you so much. That was a wonderful read and I really enjoyed that.

It's not that you get one chance to do something like this, it's that there is one total chance to do something like this. After it's been discovered, it's much harder to do it again for everyone - so if you never do it, then you don't get the benefit and Russia or USA or someone else does that and gets the benefit and you still lose the ability to do it in the future.

The second generation chips mentioned in the Bloomberg article were fitted inside the plastic of the motherboard. I don't think it's a stretch to imagine that such chips might be really difficult to spot. And if they aren't the natural next step for a nation state is to build an xx billion dollar fab factory and produce smd "resistors" that contain one of these chips.

I think the real trick is a real covert channel that's hard to find both analytically and via stastics/machine learning. Which is stil high enough bw.

It's so hard to detect that I doubt they will never do it again.

The second generation chips mentioned in the Bloomberg article were fitted inside the plastic of the motherboard. I don't think it's a stretch to imagine that such chips might be really difficult to spot. And if they aren't the natural next step for a nation state is to build an xx billion dollar fab factory and produce smd "resistors" that contain one of these chips.

I think the real trick is a real covert channel that's hard to find both analytically and via stastics/machine learning.

Especially if you:

1) happen to be country that makes all electronics

2) Company with a lot of ties to your country is happen to be top server supplier

3) Companies that use these servers happen to work for DoD, CIA and all major points of interest.

I wonder if China is making all these cheap wifi chips (esp8266, esp32) etc as backdoors into US infrastructure.

No more Chinese phones for me either way. It's enough that Google knows everything about me, PLA doesn't have to.

What phones do not have Chinese components?

Not only that, which base stations don't have Chinese components?

But cpu, baseband, ram etc is certainly more serious consideration... And I don't see how you can get around that. Unless you plan to build a dumb phone around a Motorola 68k or something?

Ericsson for one.

Ericsson do most of their QA work in China at least.

Without components from China? Today?

And how could you use esp chips as "backdoors"? Maybe espressif made them because they thought there is a good market for low-cost SoC with good network stack?

I hope that is the case; the pessimist in me says the price/performance is too good to be true.

I hope esp8266 & so are not relevant enough to have a sofisticated backdoor built in.

It seems likely that the esp8266 chip family is being used in industrial automation. It's a very powerful yet cheap chip, making it practically ideal for adding covert spyware. I love it, yet it scares me. It's a little too amazing.

It's not at all hard to imagine, especially from a country who is so paranoid about backdoors from other country's operating systems that they have written their own:


This is a Linux distro, not "their own OS". Many countries all over the world have had government-subsidized Linux distro projects, at national, state and city government levels. It's something that happens pretty quickly whenever and whereever a large deployment happens (e.g. LiMux).

For that matter, many a CS faculty at many a university have actually created their own OS. Nothing too sensational either.

Pedantry aside, it is the reasoning behind building their own OS distro that is relevant here:

> In 2009, a report presented to the US-China Economic and Security Review Commission stated that the purpose of Kylin is to make Chinese computers impenetrable to competing countries in the cyberwarfare arena. The Washington Post reported that:

> China has developed more secure operating software for its tens of millions of computers and is already installing it on government and military systems, hoping to make Beijing’s networks impenetrable to U.S. military and intelligence agencies.

Sounds perfectly sensible, rather than "paranoid".

Paranoid like Germany, France, Spain, Austria, Venezuela, Thailand, Turkey, etc...?

They all have Linux distros, sponsored nationally.


“You only look behind the door if you have stood there yourself.”

It would explain the bizarre hiring practices - plausibly relevant, but often in areas most devs just don't use day to day - that you could coach fellow agents to pass!

(I've never actually bothered interviewing with any of the above, maybe it's all entirely normal or you need to write your own self-balancing tree implementation regularly).

It's probably a lot easier to just blackmail or bribe existing employees, probably similar to how the supply chain hack worked.

The open questions seem to be:

- Is a legitimate news organization incompetent when it comes to a deeply technical investigation?

- Why are large publicly traded tech companies refuting the article to the point of potentially defrauding investors.

I don't think the power of international players and the maturity of their strategies plays into it very much. I think you can assume they are incredibly powerful/pervasive and still doubt Bloomberg.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact