Hacker News new | comments | show | ask | jobs | submit login
The Big Hack: How China Used a Tiny Chip to Infiltrate Amazon and Apple (bloomberg.com)
2493 points by Osiris30 13 days ago | hide | past | web | favorite | 769 comments

For those interested, there are additional threads discussing Apple's and Amazon's denials:




Is there an article that describes a bit more in detail what the chips actually did (or were capable of doing)? They only say "the microchip altered the operating system’s core so it could accept modifications.", which I might interpret as circumventing signature checks to allow installing modified firmware on the systems? But how does the chip connect to the network and how does it receive commands?

That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.

SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.

This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.

The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.

OpenBMC (Facebook, Google, Microsoft, Intel, IBM and others) is working on open-source baseboard management software, https://www.linuxfoundation.org/blog/2018/03/openbmc-project...

> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.

LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/

Wouldn't help. The BMC hardware has direct serial access to CPUs and other hardware in the machine. Communication is unencrypted. A hardware modification attack wouldn't touch the firmware at all and could still compromise IPMI functionality.

Reducing the attack surface does help -- you're making perfection the enemy of the "somewhat better".

No, against determined adversary it doesn't.

Even now, supply chain hardware modification attacks remind people of fiction. However, the number of people known to be affected by buggy BMC firmware is orders of magnitude larger, as described two comments upthread:

> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.

It sometimes feels like certain hardware protocols were designed to be insecure. I remember reading about IPMI issues back in 2013:


SuperMicro hardware in particular always struck me as such. Asking users to pay a license fee to be able to update the BIOS on their devices (after paying tens of thousands for the hardware itself) is a kick in the teeth.



To be clear, this is only for the ability to update the BIOS over the BMC interface, not for BIOS updates in general. (unlike some other vendors, where you need a support contract to be even able to download the updates)

At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.

Supermicro doesn't publish changelogs for their BIOSes. Their disclaimer on their BIOS download page says, "don't update the BIOS unless you have to". It's a ridiculous stance for a server company. You buy Supermicro over Dell for the price. That's pretty much it.

Price, variation, and time to market.

For the models I was looking at a number of years ago, the options were through IPMI or via a USB Floppy Drive. Perhaps my memory is failing me here, but I seem to remember being quite enraged at the prospect and would have done a lot not to have to pay the license fee on principle.

"Designed to be insecure" is probably unfair to the designers of IPMI. Security was just not as big a concern as it is today.

This is only really valid for protocols or products designed before the Morris worm of 1988. Anything designed beyond 2000 has no excuse for not thinking about internet security.

Well, IPMI isn't supposed to be exposed to the internet. Best practices have you running your BMC's on a completely separate, highly locked down administrative network.

I did a security audit on a company that had a setup similar to this.

After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.

Management networks need rack level isolation.

Best practice would be for the BMC to not have access to the regular network ports when it has a dedicated network jack. All the ones I've looked at don't have any kind of physical interlock or switch, it's a software interlock.

Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.

Well yes, typically you'd 1) configure IPMI to use the dedicated port, 2) put those ports into a VLAN with no outgoing internet access. But since this is BMC, what's stopping it from just using your management or production port to fire off its secret phone-homes and whatnot?

Computer security as a field and area of concern is significantly older than IPMI.

The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.

If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.

> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC

cool, thanks for that info.

> just to write off the hardware

maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.

at the expense of a dedicated switch.

Or, if the server has a free PCI slot, stick the cheapest possible NIC in that.

Cannot it be physically disabled with a jumper or maybe by cutting some lines on the motherboard?

No. SuperMicro make some models which appear to be for government application which lack the ASIC completely, they were available on NewEgg with no further description of purpose at one point.


But without the IPMI kernel modules loaded, IPMI is harmless, right ?

No, it boots prior/separate to the board itself. It's basically a mini-PC embedded in the board that has it's own CPU/Memory and tentacles attached to everything in the mainboard.

No, the controller runs it's own totally separate OS, has connections to most/all the buses and is able to emulate devices whether you have drivers installed or not.

No, since it can configure the BMC, which works without the OS.

The article says:

But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

To me, that makes it sound like they could download from a remote host and inject code and do literally anything.

If this is indeed the case, I'm surprised someone didn't catch something earlier when the device was calling "home" over their network. I'm wondering if China stole BlackRidge First Packet Authentication tech [1] to keep things dark. BlackRidge is... "involved" in IC and defense projects.


You're not the only one worried about Lenovo.


Not the chip itself but there's an investigation which I've already submitted:


  5. When a server was installed and 
     switched on, the microchip altered 
     the operating system’s core so it 
     could accept modifications. The chip 
     could also contact computers controlled 
     by the attackers in search of further 
     instructions and code.
So, in typical vulnerability/payload/exploit fashion, the board's bus is vulnerable by default, because the chip pierces all the usual lines of defense protecting against network and operator I/O. It carries a payload intended to target very common features used everywhere commodity servers are used, one that likely listens for DMA traffic on the bus, and alters the signal stream, by escaping upon the occurrence of a magic sequence, and inserting its own signal, before resuming the authentic stream in flight.

The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.

Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.

The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?

It's not particularly magical, there's consumer chips around which are not a whole lot bigger (though obviously in a more standard package). You don't get a lot of resources, but you don't really need it if all the other frameworks are in place in other software. If this sort of thing is something you can buy on Mouser for a few cents, the espionage grade material is probably an order or magnitude more higher quality.


For scale, this alone is about the size of a large SMD capacitor and would basically be lost in most designs today.

For another example, I have a couple of these which are a bit bigger but have an ARM SoC and onboard Bluetooth (with antenna):


That's a crazy module, I had no idea anything of that scale existed for Bluetooth radios.

Figure it's custom silicon, given the nature of the story, and "magic" in the sense of "magic number programming" to time the attack.


For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.

This sounds like speculation. I'm quite capable of coming up with my own unfounded speculation, but there is a real report out there with the actual details in that really needs to be made public, legally or otherwise. There ought to be a CVE about this. Where is it?

They didn't do anything to the CPU, what they did is the modchipped the line from EEPROM and the board management controller.

They probably found it out when they were repeatedly tried to reflash the BMC flash, and saw that checksums did not match.

That would make a lot of sense and would give the attacker a way to interface with all of the other hardware (network, disk etc.). Do you have a source for this information?

I looked up supermicro blade motherboards, and saw that the chip was right near the IPMI chip's line to spi flash.

And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"

Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.

Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.

An update on that theory: AST2400 has option for two SPI memories, one main, one "recovery."


The recovery overrides the primary if detected by default.

The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.

This is the most plausible theory I've read in this thread. Assuming the image in the article is a stock image (there isn't yet a clear image of a definitely compromised board), then the added part could simply be another TSOP8 Flash part. This implies the firmware to the AST2400 is unsigned (which it appears to be, as there's coreboot options for it).

That makes the whole thing gloriously simple. A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.

To me it looks like that bloomberg intentionally photoshoped chip labels to avoid risk of upsetting their manufacturers.

This would hardly fit the description of a chip "smaller than a grain of rice", though.

Is this datasheet available outside of CSDN? Do you by chance have a copy you would be willing to share?

eh, does this mean that the motherboard is not tampered with ?

Well, it means that provision for the second flash was already there, and PLA simply exploited that fact that Aspeed chips are virtually omnipresent in higher end servers.

It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.

So, they would've been screwed even if they were doing board testing outside of China.

That's a clever trick.

But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.

A photo of such a motherboard with a big arrow pointed at the additional chip would be a useful addition to this discussion.

Supermicro 6128 aka x10 series microblade. Those were very popular among Chinese DC operators during Broadwel era.


Left of the sata connector. An empty space with 8 pads for an smt eeprom or flash. It is occupied by the thingy on bugged boards.

Right below is the Aspeed chip - the BMC

Is there a list of known compromised Supermicro SKUs?

There really, really needs to be. If this story is real and things have been known compromised since 2015, it’s unacceptable that this information isn’t shared more widely. The “we’re going to follow them” idea doesn’t hold water over 3 years, countless small vendors using Supermicro and not being well-connected enough to know (like Apple or Amazon) about it.


Hm, but DMA messages get distributed over a parallel bus and this chip seems to employ a serial interface, so I would assume that it's not directly connected to anything that requires high throughput (i.e. memory, disk and peripheral access).

They attacked the Base Management Controller. There's an article by Bruce Schneier from 2013 warning about exactly this attack. Quoting:

"Basically, it's a perfect spying platform. You can't control it. You can't patch it. It can completely control your computer's hardware and software. And its purpose is remote monitoring. At the very least, we need to be able to look into these devices and see what's running on them."


> You can't patch it.

Sure you can. OEMs regularly release patches for platform BMCs.

Not sure about you, but I'm not an "OEM".

Well, you cannot patch the vast majority of the software in your computer (assuming you are like the vast majority of users using proprietary crap for everything). That does not mean it is all unpatchable. If supermicro care, they could release a BMC update, for example.

> They attacked the Base Management Controller.

Do you know this, or are you speculating?

It's in the article:

"The illicit chips could do all this because they were connected to the baseboard management controller..."

Can you quote a single source from that article, or is it all anonymous?

That is also in the article...

"17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."

So not a single source. Might as well be going to war over invisible weapons of mass destruction. Also, Apple and Amazon both has said they do not agree with these claims. So far this is nothing more than propaganda.

so you prefer to believe official corporate statements saying they didn’t have security leaks, rather than a news agency with 17 independent sources confirming they had ??

That’s an interesting choice.

17 independent sources doesn't mean anything. Again, this is the same as the sources that said their were weapons of mass destruction in Iraq. Companies and government use propaganda all the time, so until you have more information confirming these claims, then it is best to hold off before trying to say that Apple and Amazon denying the claims str proof that they are real.

Are you saying Bloomberg made the sources up, or that they exist but are all lying, because they won't go on the record?

Def possible they are lying. There are a lot of companies competing with white box manufacturers. If you can't name a single source, and you have two major companies saying these claims aren't accurate... then maybe you should hold off before making judgments.

This reminds me of that old story about the Xerox copy machines that the Soviet Union bought.

Where each unit was planted with a image recorder. And for years, the American spy agencies had a great laugh, that they were able to intercept all the documents that the Russians made a copy of.

Back then, this was an off-network infiltration. Where the copied images, were retrieved during regular servicing intervals by a Xerox technician.

Huh crazy, I hadn't heard about this before! Just found a couple posts on it [0][1], for anyone who's interested.

0: https://electricalstrategies.com/about/in-the-news/spies-in-...

1: https://discover.cobbtechnologies.com/blog/the-soviet-union-...

I was born raised, and went to CS undergrad program in China before come to US. Yes. I can testify that the "Xerox Copy Machine" is the first thing they teach you about information security :p

Or the IBM Selectric Typewriter implant.


One of my personal favorites.

I had never heard of this...but now I can only imagine how exciting this must have been as one of the engineers-- working on a top secret project for the CIA in an abandoned bowling alley: https://electricalstrategies.com/about/in-the-news/spies-in-...

edit: whoops, looks like amatecha beat me to posting more info

Exciting, but not so ethical. We owe society to put our knowledge towards making it better for all people, not just "our team".

It's important though to make sure that your team keeps tactical advantage so that it can continue existing. Maybe someday mankind will find world peace but in our current world there are a lot of nations that hate each other still and wouldn't hesitate to take advantage of weaknesses of other nations for personal gain.

It's equally important to question your "team".

In the developed world, hatred is often manufactured to gain power... The leaders of a nation can cause more harm in their quest for glory than the average citizen of an opposing state.

If our team is resorting to unethical and immoral ways to gain that advantage, then we can't take the moral high ground and also can't complain when the other team also does "whatever it takes" to gain an advantage.

Also, security through obscurity is, as we know, an illusion. Information always finds a way out.

I understand your point, but there should be limits.

As I tell my small human, The good guy must do only good, or he is also the bad guy.

What do you tell your small human about the current behavior of many prominent adults, or if the small one is too young to see what's happening, what will you tell them?

Good on you, man. The world needs more good guys.

But once you create the technology and hand it over to someone else, you have no assurance that it's only going to be used against the "bad guys":

Judging by the number of parts ordered from Xerox, Zoppoth believes that spy cameras may have been installed in photocopiers all over the world, to keep an eye on U.S. allies as well as enemies

+1, couldn't agree more. I wish more nations/Governments would think like this, but too many are motivated by more wealth and power. At some point, you would think, the animal would satiate itself. It appears that is not the case.

> Exciting, but not so ethical. We owe society to put our knowledge towards making it better for all people, not just "our team".

Our team promoted liberty and democracy worldwide (see below). The other promoted totalitarian dictatorship, labor camps, etc. If your team is liberty and democracy - self-determination for all - then it's not 'our team', it's everyone's team.

To address the elephant: Yes, there were many, many exceptions to the West's support of democracy, and many of them were awful (Indonesia, the Mideast, Zaire/Congo, and many others come to mind) but beyond a doubt, during the Cold War and after democracy and liberty exploded - stop and compare it to any other time in human history. Lots of bad things happened, but compare that with Soviet-dominated areas such as the USSR itself and Eastern Europe.

Like the IBM typewriters with keyloggers that the Russians planted on US diplomats during the 80s?


I think Russia became much more aware of that kind of security threats. CPUs, motherboards etc are produced internally, i.e. https://en.wikipedia.org/wiki/Elbrus-2S%2B

wow. what goes around comes around.

"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

Well played, Bloomberg. Well played.

To be fair, removing the last three words could make it sound like the Mormon church was beaming sermons to the adult film industry which would probably be even worse...

I LOL'ed when I read that line too

I laughed at that line too.

I have worked in card payment industry. We would be getting products from China with added boards to beam credit card information. This wasn't state-sponsored attack. Devices were modified while on production line (most likely by bribed employees) as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag.

Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).

They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.

We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.

Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?

No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."

I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.

There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company.

The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).

If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.

All employee actions are company actions. You partnered with a company that can’t control what it’s employees do? No internal audits to make sure their reputation wasn’t being tarnished by a few employees?! Your loss.

I understand the indignation etc etc. And the suggestion to not use these kind of companies anymore.

And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it.

One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets.

The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.)

If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.

How much do you think you would have to pay them to make it worth their while to not use cheap components (so as not to risk losing you as a customer)? It’s amazing how people never consider using economics.

Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?

As somebody else noted: it doesn't have to be the company that does it, but a lower level employee who cuts a deal. So paying the company more wouldn't solve the issue completely.

All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.

And again, you should make it the company's problem if they can't control their own employees. They can learn to put some controls in place once it starts hurting their bottom line.

> All of these things are extremely cost sensitive.

Say their profit margin is 1%. If you offer to pay them a 1% higher price, you're offering to nearly double their profits on whatever they sell to you.

Admittedly, that only works if you are a sufficiently large portion of their business. If you're a tiny percentage, then it may take a lot more to motivate them if their other customers aren't willing to pay more.

If the solution to the problem involves line checks, independent audits, personnel checks, increased pay, etc., thenyes, throwing money at the problem is a large part of the solution.

You have to pay them as much as it would cost to manufacture in the Western country doing the out-sourcing...so unless the cost of these extra quality checks outweighs the cost of on-shoring you might as well measure angular momentum.

Madness if you ask me but there we are.

> I understand the indignation

IMO this is not indignation, it's supplier sourcing 101.

> If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.

I will definitely agree that holding suppliers to standards regarding consistent output, unadulterated products, and conducting audits all make production much more expensive.

But if you're a device manufacturer, these sourcing controls are key to shipping quality products.

lol. you haven't worked with a vendor in China before, have you?

But make no mistake, this yet another globalized race to the bottom, but one that hits capital (or the end consumer), rather than labor or the environment.

>The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty

Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).

I used to work on the server-side stuff for telecom devices. We designed hardware that went into customer homes, but only downloaded the (encrypted) firmware upon home activation as otherwise the Chinese manufacturers would have ripped us off and sold them to telecom companies without our cut.

So, realising they could copy our hardware, but didn't have our software, they responded by trying to hack my servers, multiple times, from the same IP they sent manufacturing data from. A quiet word with their management would stop it, and it'd start again a couple of days later.

These people have no shame, and if we are going to go for lowest cost at all times it is what manufacturers should expect to happen.

OMG, that is so so so so brilliant! I'm going to go tell my boss right now that the suppliers better suite up, otherwise we are going right to their more expensive, less experienced vendor and will delay our product launch for a year....and likely still suffer the same problem. Those Chinese vendors better shape up or we're going to really teach them a lesson by driving ourself out of business right quick!

Just imagine for a second if someone was advocating on HN for the use of a web framework that has knowingly allowed itself to be compromised to steal passwords because it was cheaper to use.

How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.

You mean like couple of well known platforms that let you reach their users and in the meantime steal, mine and monetize user information?

So no more using Intel?

Leaving an abusive relationship tends to be scary, especially if the victim is convinced that the abuser is the least bad. Doesn't mean the thought of ending that relationship should be dismissed out of hand.

You can be as sarcastic as you want, but these stories are absolutely indicative of a much larger problem. It might be beyond the capacity of one company to fix, but in the aggregate they represent a serious political and economic threat and need to be dealt with one way or another. Preferably by literally anyone other than Donald Trump.

Speaking as someone who is _not_ a citizen of the USA and is well aware of his issues, he seems to be first US President who actually appears to have the guts to do something about it. I doubt an establishment politician would ever have imposed tariffs on 200 billion dollars of China imports.

If he stuck to just China I would agree. But by trying to change up numerous different agreements, he lost the goodwill needed to have a collective crackdown on China. Even if he is the first to be willing to pick these sort of battles in a long while, he has a new issue of picking too many battles. Seems like a Goldilocks problem.

The tariffs aren’t the problem, it’s the stupidity of going it alone and alienating allies that could have turned the thumbscrews with you on China to get them to really capitulate.

> some employees were doing it with no involvement of the company

Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.

Also it comes across a little naive. Oh they told us it was just an employee and would never happen again, OK!

>There were other considerations like the fact we were actually buying it from large reputable company and what happened was that some employees were doing it with no involvement of the company.

It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way.

Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?

the indignation was not that. you are still missing the point.

after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.

Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.

> Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.

I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened.

However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.

PCI DSS allows for "Mitigating Controls" if you need to deviate from specified requirements, provided it is well documented and is equal to or greater in security. Doing teardowns to review circumspect hardware, and applying one's own tamper protection deal (and with accompanying documentation and tracking/logged information) would very likely be sufficient to maintain complaince.

You all recognize the irony, right?

"There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company."

I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.

>If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.

This still sounds nuts to me.

Two thoughts:

> they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them.

I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs.

And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify.

You could also be mandating how many boards are allowed or that the SNs go on early in the build process.

This is just how low trust societies function for hundreds if not thousands of years. This is also why families tend to be much stronger in these places. The American idea of a strong family is different from what I'm describing, since it just centers on immediate family. My definition of strong family bonds, implies both closeness and dependency not just on your immediate family, but even on your distant cousins on both sides. It's close if not the same to how Mediterranean and Hispanic cultures view families. I guess you can call it a clan mentality. I'm now wary of countries where families are really strong ie clans. It's not always the case, but it tends to mean the rule of law is weak and corruption is crazy.

When you can't even trust that food is real, and when you have to bribe even low level provincial government employees (can you imagine needing to bribe DMW workers or even police?) - what else is an average person to do except to treat it as the norm in order to survive? Unless you're an elite, the only other option is to leave, and not everyone has that choice. Very little is considered wrong over there, aside from criticizing the powers that be; as long as it helps your clan. It doesn't help when Western companies and governments look the other way, as we see in past news stories and even the comments here.

I'm not condoning the behavior; just explaining.

Not much to add, but want to note that you're spot on. I've encountered this SO many times over the years and this is always the explanation I'm given. At first it just made me indignant, too, and to some degree I guess it still does, but now I just try to let it go.

It's not rewarding bad behavior when you take it into account in the deal. Moralization is useless when you do business with different cultures. When you notice something what would be out of ordinary in your culture, you estimate the cost and find out how locals solve it. Different cultures have different kinks and people living there know to adjust to them. It can be fun to figure out how other cultures function.

From a naive Finnish perspective you could see American business culture as dishonest. People lie all the time and promise things that are false and you need massive amount of legalese or they fuck you over. When they say something, you can't trust it as much as you can trust a Finn (or Swede). American way is to lie to your face in pre-negotiation but then be truthful to the legally binding agreement. Germans or Belgians are more truthful beforehand and they stick to deals better without legal enforcement looming over them, but require more detailed written deals than Finns. Who is right? Should we punish Americans for their behavior or adopt their culture?

Of course not. You must understand that that the level of acceptable exaggeration, overpromise and bullshit and trickery is strictly culture dependent and relative. Someone is dishonest if they "lie this amount above average" in their cultural context. After you make an adjustment, you know how to get the truth.

Chinese manufacturers have the capability to stick to standards, take responsibility and deliver high quality stuff. It's just more dependent on personal relationship. Just placing an order has less trust as default. Either you develop "quanxi" with your suppliers or work trough some third party that understands both worlds. Third party understands what you need and knows how to get it from the Chinese suppliers because they have a relationship.

Chinese pay the cost that comes with their business culture. They can't network as fast as you can in open western cultures with more default trust between strangers.

Altering components to something cheaper is common practice for every company that is willing to exploit the fuck out of their customers. Look at the food industry. In EU there is control, in US thare are sanctions. But it is up to the governments because companies are to opportunistic to change. As long you can (re)sell the junk you get, you're fine, if you can't you upgrade "QA".

Both the EU and US have internal controls on food quality and sanctions.

Contrary to the impression your post gives, the EU has much more "sanctions" against food (esp. imports) than the US does, and is generally taxing much heavier and restricting many more things. Importing essentially any milk product into the EU is only possible on an exception basis. Trump has a LOT more work before extra sanctions in the US will become comparable to the EU ones.

What is often not said is that the EU not only has sanctions on its borders against food products, it actually has internal sanctions and regulations that are purely political in nature.

For example "champagne" in the EU can only be made within a particular region in France. Which leads to funny, in a sad way, consequences:


This applies to many products. Wines, cheeses, cakes, fruit, meat, potatoes, candy ... and is generally absolutely ridiculous:


For your other part of your comment, I am unwilling to give a real answer. Obviously the US checks it's food quality. It's made different choices than the EU, famously the US allows hormone "enhanced" meat, but it does check food quality and safety. There are heavy penalties for breaking them.

Champagne is literally the name of the area where the product comes from. It makes perfect sense that anything that doesn't come from the region Champagne, isn't champagne... Call it sparkling wine, it's fine, but not champagne. And it applies to most local products. Camembert? Comes from a specific town, which is called Camembert. Beaujolais? The name of the province. And the list goes on. It doesn't sound "absolutely ridiculous" to me.

"generally absolutely ridiculous"

Did you look at the map? It's nearly all things with a place name in the product name. For example, "Welsh Beef". Why is it reasonable for a consumer to buy Welsh Beef and for it to possibly have come from somewhere other than Wales?

It seems to me, more like generalising the concept of trademarks to place names rather than just company names and brand name.

and no kinder surprise in the us.



Conflating culture with race is ... racist.

Not all cultures are great. If you are claiming “all cultures are great”, then perhaps “great” has lost any useful meaning.

Discrimination, as the ability to discriminate features to identify useful pattens, is not *-ist; it’s a critical skill that is being sullied by some neo-intellectual BS that passes for “equality” or something.

The ability to identify patterns of behaviour prevalent to a class of suppliers and avoid them, rewarding other supplies — even from the same area of the world — who are better; even “great” perhaps, is a valuable skill...

But perhaps I’m wrong. They’re all great. Carry on.

Give one example of a non-ethnic Chinese person who is considered to be culturally Chinese.

It's the norm in the west to not conflate culture with race. But to think that this is the norm in the rest of the world is to project your value system on others.

Culture is not equivalent to race. Cultures that permit slavery, for example, are not that great.

I had the same initial gut/emotional reaction, but after a re-read, not _sure_ if it's justified.

If we define "a culture" as set of sociological customs, value systems, expected behaviours and stimuli/response criteria; and if we define "culture greatness" as such sociological customs, value systems, and behaviours that lead to specified and asserted goals;

than one can reasonably compare and contrast cultures in meeting (or even striving) to these goals, and thus discuss their greatness.

Only to the extent that the relationship between race and culture is causative rather than correlative.

If forced at gunpoint to trust one person in my supply chain, I would instantly choose the Taiwanese-American in San Jose over the mainland-Chinese in Shenzhen, every time. Likewise, I think exactly the same Indian employee will be more productive if physically transplanted to Silicon Valley and surrounded by assimilated Indian-Americans than if set up in an office in Bangalore surrounded at all times by other Indians and Indian businesses. And equally likewise, I have noted a large difference in my own work productivity doing similar tasks under different employers, or even in different work groups under the same employer.

So it isn't even national culture, racial culture, or regional culture, but also corporate culture. Breaking news on the Obvious Channel: social apes found to be heavily influenced by their peers.

Ancestor post was obliquely referring to the oft-reported tendency of mainland-Chinese manufacturers to try to cheat their customers. They have been caught copying protected or trade-secret IP, counterfeiting, substituting cheaper out-of-spec parts, and inserting hidden hardware backdoors. Outside of China, engineering/construction companies tend to import Chinese subcontractors rather than use any local labor. That's all just PRC-Chinese business culture. If you pluck individual employees out of a Chinese firm, and hire them into a multinational based out of another country, they will be perfectly fine, aside from the slightly elevated possibility that PRC intelligence will aggressively recruit them as an asset to steal exploitable business secrets. If you can manage to pull everyone they care about out of the reach of the PRC, they will likely be a great investment for you.

Clearly, the national-political-business culture of mainland PRC China is hostile to businesses that have significant interests outside of China. The government there is racist, controlling, and mercantilist-protectionist, and the businesses with a physical presence have to fit in to that culture in order to exist, along with the traditional system of guanxi social capital. It makes them difficult to work with, but as long as the price is right, it's still worth it to try. You just don't give them access to any secrets, and make sure your QA department is full of no-nonsense hardasses. And as mentioned elsewhere in this topic, that means you may have to measure total mass and moment of inertia around multiple rotational axes in order to find any unauthorized modifications that might expose you to company-bankrupting liability in your business culture. You can't "trust, but verify"; you might just have to check every unit of every shipment.

As for culture culture--wushu martial arts, the action movies including it, silk, noodles, jadeite carvings, fine porcelain, traditional architecture, religions, herbalism, the woo-woo pseudo-religion attached to the medicine traditions, etc.--Chinese culture is indisputably great, especially considering that it's what's left after the PRC government tried to annihilate most of it as part of their revolution.

culture != race

> You're starting to realize that not all cultures are that great.

What measures a culture's greatness? How can this value be empirically deduced?

Honesty and trustworthiness could be two such measures.

There are many important and worthwhile discussions that our empirical tools are not able to significantly contribute to.

That's not to say that there is a such thing as single-dimension "greatness" metric. However, outlawing value judgments is going too far in the other direction. Of course, such things can be quite sensitive, and should be done thoughtfully.

breadth, quality and equal application of civil rights.

the value of civic participation and honesty


It is called business. The company's ONLY duty is to bring profit to its investors and management is legally bound to maximise it. Now, the definition of profit may differ as well as what is profitable and what is not, but the company is legally obligated to work to bring profit and if you throw a good deal because you don't like it you better explain it to your investors.

Whether you like what your supplier does or does not is of no concern. If couple of units can be detected and written off it is treated as cost and you move on to decide if it is still profitable and whether you can get better deal somewhere else.

You let you emotions rule and it just means you are not fit for the job.

Sounds like a libertarian oasis.

I am reading it as "we don't any any option outside China and we have to live with this". I guess even after adding cost of doing additional of products you buy from China its less than what you would pay to buy it from other place.

Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.

he's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive.

I can't believe the parent comment is so brazen. It makes me physically sick.

I guess it's news to you that the financial sector and especially banks in fact do exactly that:


But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.

like shoplifting. Most large chains know exactly how much "loss" they experience, and they just raise their prices to cover it. Theft and fraud is a part of life unfortunately...

knowing about it and willfully facilitating it are two different things

This problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.

Honestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one?

Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.

Maybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)

A good read is 'Poorly Made in China'.

The product is different, but the problems are the same.

* They drop changes and problems at the last minute, so you're over a barrel with your customers.

* Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise.

* Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far.

* The new supplier will 100% play the same games.

* Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way.

* Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business.

* Are you going to admit to your customers and bosses that your products were faulty and you knew?

It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.

> * Are you going to admit to your customers and bosses that your products were faulty and you knew?

If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical.

When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice.

Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure.

All this tells me is that Made in the USA is more valuable than I once believed.

I think you’re right about the value of Made in America and societies where honesty (even about problems or risks) is valued.

The book I referenced included a number of instances where factories actively undermined individuals so that they had very little choice than to either go along with the situation, or press the ‘nuclear’ button and scupper the entire arrangement, including their own livelihood and career. That’s a tough choice for people to make.

Not having the backbone to reveal it when you discover a fault is not what that comment was getting at

Profit. Changing suppliers is going to cost money.

Or: you can expect this kind of shit from pretty much any supplier. Even if you go to a western company, they all source their components from China anyway.

Probably the cost per unit is too attractive

Also it forces their customer's customer to essentially subsidize production cost.

Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage.

As a result this immoral company externalizes aspects of their cost structure to greater society.

This is also such a good anonymous story that I'd give decent odds it's made up.

He's not anonymous, though.

I've heard very similar stories from people in the card industry before. I don't think it's made up, unfortunately. Card readers from China are systematically untrustworthy.

The fact that they didn't just run the units through an xray machine would seem to support your theory.

Using the technically incorrect term "angular momentum" rather than the more correct term "moment of inertia" made me think the same thing. I would think a person tasked with building such a device would know their physics well enough to use the right term, but I may be wrong.

Maybe he is afraid of giving too much info away. By mentioning it, now the opposing side will be measuring angular momentum.

So now he needs to make a new test. Ow that that is out of the bag

> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.

You mean moment of inertia, not angular momentum.

You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.

For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.

In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.

Another implication of the parallel axis theorem is that the attacker could perfectly mimic every moment of inertia by shaving plastic. They wouldn't have to know which two axes were being tested because there are only three real numbers worth of information in the system to begin with (once center of mass and total mass have been dealt with.) In the whole MOI tensor there are only six free numbers which sounds like a reasonable number of parameters to fix by adding and removing small amounts of material.

What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!

> What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start

You're assuming that there were feasible alternatives; from their comment below:

"wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China. ... Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device. ... Fortunately this only lasted for few months until it was dealt with"

Business lesson to include a 'no China' if the hardware handles anything sensitive, with a large fee for violating it for the middle man servicing the contract.

Another problem is anti-tampering measure is applied before the initial tampering check have been applied. Anyway, that wouldn't solve everything considering that if the chip itself is tampered with, that's undetectable short of an electron microscope analysis and even that wouldn't solve the problem of backdoor in the original chip design.

As you say, that's un-winnable. The only way to really build trust is the capacity to sue your manufacturer to oblivion - I mean the real oblivion (destruction of shareholder value) with criminal charge for the company officers, not the lame single digit percent of a single year of profit with a discount if you settle.

> the real oblivion (destruction of shareholder value) with criminal charge for the company officers

This very rarely happens even if people have got killed. Also, good luck suing a Chinese company in China.

I reckon this could be addressed with contractual guarantees in exchange for doing business in the first place. Make it a contractual requirement that the supplier agree to adjudication in the United States in the case of security related concerns or other tampering with the final product or with the designs. They could still refuse if something happens, but I reckon at that point you can just work to shut them out entirely from the US market.

At this point, I think it's fair to establish an industry watchdog group that works as a clearinghouse for knowing all the American companies sourcing parts from the PRC and which companies they source from. This watchdog organization would be responsible for blacklisting suppliers from the entire US market if they don't agree to adjudicate in the US or if they fail to meet that contractual obligation).

This won't bankrupt them outright, but it will sever them from the US and if this watchdog group expands to aid non-US companies, could help shut them out of the entire global market.

I would also require registration of all top executives of these corporations so that you would make the punishment sticky if they moved to other companies. Overall, you've got to set a high punitive cost to non-compliance and non-cooperation.

You can't lock China out of the market without diplomatic implications. Even assuming you managed to get a purely private boycott of China together without somehow violating anti-trust law, by the time it started making a serious dent in the market the Chinese government would have started making representations to the US State Department. Where it goes from there is anyone's guess.

(Ironically the other direction prevails in regard to Israel: there are anti-boycott laws! https://www.lrb.co.uk/v40/n14/amjad-iraqi/short-cuts )

I honestly did not know about that. I thought that if you move any mass (remove non zero mass and place it somewhere else) there must be at least one axis which you can use to detect the change in moment of inertia.

If the mass had to all move to one other place this would be correct (the center of mass would have to change), but the attackers are able to move the mass to multiple other places.

whatshisface is correct.

Ah, you are right. English is not my native language.

I enjoyed your comment. But you are looking at the difference in the inertia tensor of an adulterated board from a non-adulterated one. Not the inertia tensor itself. Signal to noise problems with the measurement.

Indeed there are only a finite number of moments. If this was not true you could effectively 3d scan an object by just measuring its inertia response.

Seriously, why are we still outsourcing chip manufacturing to other countries? Sure it's cheaper, but we sacrifice a lot to have a society of corporate slaves build our tech. Security, core domain knowledge, capability, corporate secrets, patent rewards and enforcement, etc... All of it you throw away the minute you ship your manufacturing out of the country. I've seen enough board printing machines out there to start working on our own. As a country, we need to close this gap, more automation and capability and there will be no need to outsource circuit board printing and manufacturing. We will be much better off.

Canadian steel is considered by this administration to be a national security risk. But Chinese made boards and chips installed in weapons systems and crucial data centers? No problem.

Let that sink in for a moment.


The thing being called a national security risk is a lack of domestic production capability. No one is saying Canadian steel is sabotaged or something.

Yes, obviously- but one of our closest allies shorting us on steel supplies in a time of crisis has never happened, while espionage from China-produced tech is happening right now. Which should be our priority?

Though it is, because it happened with Kobe Steel. (Not saying Canada specifically.)

They faked the data on what it could handle; and with things such as aircraft, lives depend on that data.

That isn't a Japanese steel concern though, that's a steel manufacturing concern. U.S. Steel could do the same thing.

But they are saying that Canada would cut us off from their steel production in a time of need, which is a pretty absurd assessment of the situation.

I love a good Trump bashing moment as much as the next guy, but this is inaccurate. The DoD has stringent requirements and quality control procedures in place for their chip procurement. Not to say they couldn't be improved, but the DoD has been aware of this threat for a while, and seems to be mitigating the risk fairly well.

Also, as far as I understand the argument, it goes beyond "Canadian steel is a national security risk". A couple of years ago, Mexico was caught laundering $2B of Chinese aluminum to avoid US taxes.


The theory, from the Trump crowd, is that Canada is also engaged in similar shady dealings with China. If true, that would put the US at risk.

Even if true, how does that make it a national security risk?

If the source is China, in a hostile period they might stop sending it. But according reportings I have read, the US DoD only uses about 0.5% of steel in the US, and only ~30% of US steel is imported.

Their argument is that you need strong domestic steel industry to build tanks, ships, etc, in case of war. Not saying I'm agreeing with it just pointing out the stated rationale behind the tarrifs.

The concern was the depression in prices was cratering local high-quality smelting capacity.

However, the DOD issued a memo (penned by Mattis iirc) indicating that there was no supply related concerns. The lack of aluminum used to justify the tariffs was in fact just the result of the LME's anti-market-tampering rules creating an incentive for metals traders to stockpile the material in private stores rather than in LME warehouses.

There was a nice article about this yesterday, but I don't have the link.

It doesn't. They claimed that so they could enact the tariffs, otherwise it would be a WTO illegal tariff. I think Canada/others are arguing that it is not a security risk and therefore is indeed an illegal tariff. This is what I remember from some articles. Please correct/elaborate.

Maybe because it undermines fundamental US business interests, and therefore wellbeing? If so it still seems like a stretch to me. Doubtful we'd have no where else to source steel if war or disaster struck.

Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?

The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.

The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.

>Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?

It's certainly easier to enforce laws and observe manufacturing processes at domestic factories than it is at factories thousands of miles away in a country that actively encourages IP theft and other wrongdoings, don't you think?

Not really, no. Why do you think it be easier to police? In concrete terms: who do you expect to do the policing? Because this is the kind of work that requires an expert, to catch an expert, messing with a product created by people 100% ignorant of _what_ they're making. Just _that_ they're making them. Where are you going find enough experts to check the work of even a single production line?

Because "thousands of miles away in a country that actively encourages IP theft and other wrongdoings". It makes perfect sense. If you don't think that, try to run any business in another country. It's just obvious that it would be easier to do domestically. You just list other set of problems. Those problems only get compounded if we'are talking about another country.

Not all societies are equally corrupt.

But no societies are free from organized crime. And last time I checked, no western country was free from card skimmers either.

You can prosecute someone in your home country and even change policies with enough political support. Best of luck doing this in China without getting 're-educated'.

But note that as always: that's too late. By then, the damange has already been done, and the person you take out of the system is simply replaced by anyone else. Card skimmers sure as hell haven't gone away just because we wised up to them: we now just pay A LOT MORE on law enforcement, and all that gets us is law enforcement now at least staying in lockstep with the skimmer's increased sophistication.

Now imagine how much it'll cost to stay in lockstep when the level of sophistication becomes "a single component the size of half a grain of rice".

"Free", no. Much less affected, yes. Organised crime in Mexico and Italy is a much greater problem than in, say, the UK or Germany.

You're letting the perfect become the enemy of the good here

Not really: I'm letting the practical dictate the process. Catching this kind of inflitration requires experts to scrutinise the work of people who know nothing about what they're making, to catch experts with a much more vested interest in success, and you aren't going to find enough experts to take that job, at a good pay, to check the output of even a single plant. Let alone an entire industry.

If the question is "why don't we move chip production to the US, so this doesn't happen" the answer is "because you might be able to do that for one or two plants. And it'll probably drive up the price by an order of magnitutde, so they'll go out of business soon after. And you sure as hell can't move the entire industry because of prohibitive cost and shortage of manpower to do the bit that you want tacked on that can't be trusted to be done abroad".

It's hard to compete domestically against low prices caused by China's completely different standards for wages and human rights for laborers. Plus, here we don't like the idea of manufacturing industries being subsidized by the government (except in the case of "defense" of course...), while China obviously has no qualms with doing so.

> except in the case of "defense" of course...

Well, this thread is exactly the reason we subsidize the defense industry to make sure things are produced in the US.

>Seriously, why are we still outsourcing chip manufacturing to other countries?

I actually agree it might be better for the Americans to manufacture things in America - especially things used in critical government systems.

But this seems like a human problem - if all the factories were moved to America, couldn't those factory managers etc also be bribed?

American manufacturing in the 60s was rife with unions w/ ties to organized crime.

Business culture is different in the US and China. If you tamper a chip in China and go unnoticed you are considered smart & the winner. If you do so here you'll be getting a more negative reaction.

Western manufacturers have a tendency to buy from the cheapest supplier, even if it isn't physically possible to manufacture at that price point without cutting massive corners or making profit off of some externality. However, when Chinese manufacturers build their own brand, you'd be surprised how the business culture suddenly creates a different set of incentives.

Trust alignment of incentives, as opposed to hey-they-promised-an-actually-impossible-pricepoint-on-paper.

I think I saw this on another thread. You're saying if someone bids that they can do something for a price that's not reasonable, it's my fault for believing their mischaracterization? "Fool me once, shame on me."

I much prefer the other mode of operation: "Fool me once, shame on you. Fool me twice, shame on me."

> Sure it's cheaper

That's enough, full stop, say no more. The other costs are real yet they're either not marginal, are borne by others, or both.

You're conflating a few things there. Whilst companies like to make noises occasionally about saving the planet, doing the right thing, making (your country goes here) great etc, it's just horseshit. They exist to make money. That's it.

First, wow this is both incredible and crazy! Both the China-side hacks and your side's anti-hack. Mind. Blown.

Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?

This wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China.

Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.

Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.

How was it finally "dealt with?"

They got better at hiding and his company couldn't detect it. Problem solved.

I wouldn't know. We were just buying the stuff.

> Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?

I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?

America has fabs, both old and leading edge, but ask industry giants like Gemalto to even bother to manufacture chips anywhere outside of Taiwan, assemble the final product outside of China.

They will never do that, because they look for the cheapest solution.

The bigger the company, the less it cares about things other than cost. This is why Mediatek and Broadcom can usurp the market of network SoCs, while making products with atrociously bad support. I personally dealt with both, and say that they wholely match their popular culture image.

I don't know how it is with USA, but for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres, for as long as they get them made inside the country.

Gemalto manufactures and assembles many, many products in Europe. The entire European security cluster won't use anything else. I know because I played a part in smart card development there. And all of that was from Europe. Wafers, chips, holograms, mag stripes...

What about Japan? I know they've lost most of their semiconductor business as well, but they still have some capacity no?

I’m under the impression that China does not make chips, but they do final assembly cheaper and faster than everyone else.

I don’t know if any companies do PCB manufacturing and assembly outside of China in large numbers.

Not exactly, PCB assembly is super cheap everywhere thanks to propagation of chipshooters, what makes the cost go up is logistics - what do you do after you populate the board for your part? Ship it across the world, or to another factory behind the corner?

That largely depends on the size of your board and the total number you want to ship. As soon as you reach full truck loads or full container loads that additional shipping cost is marginal on a board level.

But still, that complicates the whole operation immensely in comparison to "just go next city block"

You are right in a certain way. What increases with distance is lead time and as result inventory. Both of which are off set by volume. Cross-border adds customs issues. Complexity is not that much of a problem nowadays, maybe it never was. You are right that all of this oblyakes sense with large numbers, small scale production is better done locally as a rule of thumb.

Sony production lines assemble Raspberry PIs in the UK:


That is true I think because of what happened to ZTE earlier this year.

I don't know that.

> for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres

Well, if your chips are bigger and slower, you will need more chips and mounts/packaging to place them. If you need more chips then the weight of missile/plane/tank will be increased and available space decreased.

So at the end, the 'uncaring' military will receive a weapon which is worse than competition.

All thanks to an outdated chip.

It's not the chips that are the problem. Most of Intel's fabs are in the US, and their assembly sites are in a number of countries.

I think people should know how stark the differences for assembly in the US are vs outside of it. Something that costs, at low prototype volumes mind you, $20 in China for a dozen boards or so, would run hundreds of dollars in the US and still take the same amount of time. As it scales up, the ratio might improve, but these aren't like 10%-20% differences.

This is not microchips, this is basic PCB assembly

But I'd guess momentum is hard to change.

Offshoring has its costs.

It's the unknown unknowns that get you.


Nationalistic flamewar is not welcome here; neither are political or ideological flamewar. We ban accounts that use HN primarily for those things. Please read https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here.

> Has the West completely lost the ability ...

at first I had the same thought. but i have to question how securely the same manufacturing could be done in a US plant.

the US employee base has its fair share of desperate, ethically challenged individuals. and plenty of incentives to make a quick buck could be offered here too. idk.

The consequences if US citizens or residents get caught engaging in espionage for a foreign government are go-to-jail-for-years serious.

And it's not white collar resort prison ... no no no!


ok. a fair point.

OTOH, given US law enforcement's low efficiency what are the chances of being caught? and what if it's merely corporate espionage?

finally, the US is an open society with strong personal freedom guarantees builtin. what if the perpetrator has ties to a foreign country and simply leaves the US after they've installed the vulnerabilities?

Death penalty.

You are underestimating the FUN of playing anti-anti-^N-hacks. I have had the privilege to be paid to so anti-anti-^N-hacking on a firewall thingy in the past and it was a challenge and a joy!

The day I figured out to measure the angular momentums and calculated the feasibility I was walking around the office proud like a peacock.

Honestly if it weren't programming I feel like this is a movie-worthy story. To me it sounds so thrilling like a spy movie plot but am I just imagining that or was it actually this crazy / cool / integral to way bigger moving parts / things like I'm assuming? Regardless kudos. Your story definitely started my day on a happy note, thanks!

The fact that you keep referring to moment of inertia as "angular momentums" make me doubt that your story is true.

English isn't my native language. Of course you are correct it is called moment of inertia.

how would you measure the moment of inertia though? probably by checking your changes in angular momentum when applying a moment

^ This gave me a much needed smile today, thanks for sharing.

I'd very much love to hear more stories if you have any!

We had MasterCard end-to-end test auditor on site. This is the first time ever you get to do a transaction with real transaction system with real credit card.

Due to requirements we opted to have the only large meeting room to have outside our secure zone. This created an issue as we had no network access from there and in the end we decided to use slow GPRS terminal for the test.

The end-to-end test starts with offline transactions which by their very nature are quite fast (it is negotiated between terminal and card).

But then we went to online transaction and it finished instantly too.

The auditor, bewildered, proclaimed the test failed as he assumed it was incorrectly processed offline instead of going online. But then I pointed out to the printout to show ARQC (basically says it was certified online).

Now, the real discussion started. The terminal was very slow taking quite few seconds to establish GPRS and then even more for the SSL handshake so the auditor said it was not possible to make it work.

How it worked was that I have completely gutted OpenSSL and had entire cryptographic state stored locally (safely, using internal HSM) so the SSL session could be optimistically re-established without another handshake even after TCP connection was closed. The first message the terminal sends is already encrypted transaction message, there is no SSL handshake. I wrote an application to terminate the connection in our data center so that it stored the states of each connection in the database. The entire handshake was only done if the first message could not be decrypted successfully.

The operating system was single-threaded with no multitasking of any kind. This meant that all applications on this device did their operations sequentially. Send network message, print something, display something, etc.

I wrote a cooperative multitasking functionality into the application (using coroutines) so that it could work on multiple tasks at the same time (like talking to network and printing).

I then have segregated all data on the printouts so that it can start printing without having to already have response from network. Hopefully if everything went right, the response would come before it even came to that place on the printout effectively looking as if it was done in zero time.

FWIW, the described technique (or something roughly equivalent) is now standardized as 0-rtt early data in TLS 1.3. (you still need 1-rtt for TCP, unless you can combine this with tcp fast open, or run TLs over UDP)

Seriously, pitch this story to an editor at one of the major tech blogs. It would be an incredible read.


But am I wrong to have my hackles raised by a) the roll-your-own security nature of this, b) the reliance on a single developer's single stack implementation as what guarantees the integrity of the system? It seems like there are a lot of assumptions baked in.

I, too, would love to see a more detailed write-up--if there's a big idea here (almost a unikernel thought), it deserves to be shared and tried by fire.

Agreeing with mortenjorck. I'd love to read longer stories.

i love this ~

quote captures the human element playing strong in face of bad system

a good hack can sometimes be it's own reward

When I worked in telecom (a while ago) the manufacturing was shifted from China to Thailand/Other SE Asia due to this. The Thai companies weren't as efficient, but were much more open and honest when problems would arise., plus they didn't blatantly steal tech

I think what is described is an issue with process.

If the device is sealed with an anti-tampering system then the contents must be checked by a trusted entity before being sealed.

Trying to guess the contents of a box that you cannot open sounds a bit like madness.

See, the article showed that even largest companies are not completely immune to the problem. This was decade ago and payment card industry, not exactly national security matters.

The PLA can lean on factory managers very effectively but they're not going to be interested in small time stuff like credit card numbers. The sort of sophisticated criminal gang doing something like this will have fewer coercive tools at its disposal and I'd imagine would target lower level employees with bribes.

I'm not sure; the DPRK are certainly known to make their espionage units self-funding through credit card fraud.

The problem highlighted in the article is very tricky because verifying a motherboard to that level of details is difficult.

As I hinted in another comment I suspect that they had a suspicion and checked those motherboards very, very carefully.

> guess the contents of a box

Use X-ray? or whatever can penetrate the exterior shell

All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia and these are quarantined immediately after they return. They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).

The more critical the field, the more you have to treat those devices as untrusted before attaching them to your trusted zone.

> They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).

Given how sophisticated these attacks can be, I'd think they'd issue disposable equipment to be destroyed on return, like a cheap netbook or something. I don't see how you could trust an individual viewing a simple X-ray scan to detect some extra microchip the size of a signal conditioning coupler.

Procedures change, as attacks get more sophisticated the next step could be disposable devices. But an attack like the one described in this article won't be mitigated by having a disposable device. On the other hand having your laptop "hijacked" while on a business trip will most likely involve some extra PCB or components that are a little more obvious that something that's "built in".

Then again many companies or public institutions would find it hard to justify shredding each week maybe tens of laptops and phones that still have to be good enough to work on. Basically they still have to be a "standard issue" device with your company's software stack, config, etc.

I'm sure someone can find a good compromise between security and wastefulness.

The obvious solution would be to earmark some number of dedicated "china laptops" that will never be trusted with anything important, and get reused everytime someone has to visit China. If they get backdoored... who cares? They can only spy on things that were getting spyed on anyway due to being in China.

Does the security team have any motivation to reduce waste? Unless you're finding hacks in your devices already I see no reason to think they aren't just making you jump through hoops because it's funny.

Of course not. But remember that someone else is holding their purse strings so they might care about that.

The post-trip inspection is not so that the device can be reused, it's so you can (try to) find out if it was compromised. A $3000 laptop is not a significant cost compared to the airfare, hotel bills, etc.

But it's useful to know when/if you're being targeted.

Add the USA to that list.

Are you saying companies should or that you know of companies that do?

As far as I know, big EU companys do that, when they visit US, or they don't have sensitive information with them in the first place.

The problem is not the data on the machine, it's that you connect that machine to your trusted network when you're back.

Especially when border cops can seize your phone / ask for your passwords

One of the companies named in thr Bloomberg article does. They just deatroy your laptop if it was in the hands of customs without your supervision for any length. US customs explicitly included, which is kind of wierd if you ask me.

That’s perfectly expected. It’s not a stretch of the imagination to think border checks are abused for industrial espionage. If it gives your country a major advantge nothing is off limits these days.

That doesn't strike me as odd at all.

My employer - doing lots of interesting engineering in among others the marine and aerospace sectors - certainly do that; we're based in Europe.

Basically, any device brought to the US and a number of other countries are issued for one-time use; if they leave our custody for even an instant, they are to be scrapped.

Some companies certainly do that (in case the question was for me).

> All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia

From what I understand, Boeing does this when employees visit France.

They have had problems with men in nice suits going through laptops stored hotel safes.

And none of those measures would have protected against the compromise detailed in the article.

Sure, we were discussing someone saying their devices came with extra PCBs inside (it's a bit hard to follow but scroll up to the original comment, currently first on the page).

> We would be getting products from China with added boards to beam credit card information.

>> Trying to guess the contents of a box that you cannot open sounds a bit like madness.

>>> Use X-ray? or whatever can penetrate the exterior shell

2 different types of attacks, 2 different types of responses.

I would imagine you would treat any kit that has been to those countries as disposable on return and crush them

Yes, irrespective of country where its manufactured, if there are compliance requirements around an un-openable box, then some process becomes required.

But I think the GP's question is: "Whether it would be cheaper" - in the sense whether such an expensive QA process could have been averted by having a more trustworthy partner. One whom you're not on a race hack after hack.

The point is that if the devices are sensitive with compliance requirements then you must be able to verify them irrespective of who you hired to manufacture them.

You cannot just trust the word of a contractor on this because it's your ass on the line.

The point is that the process was to assure the device wasn't tampered AFTER shipped from manufacturer. Nobody thought it could already have been modified so early in the process. This is the eternal cat and mouse game. When I started in IT in 90s it was assumed that company network was quite safe and you didn't always need passwords, maybe for critical resources only.

I would think that, logically, and as illustrated, "the device wasn't tampered AFTER shipped from manufacturer" means after YOU have shipped it to customers. The anti-tampering system is to prevent modifications in the field.

The manufacturer shipped the device to us from China. We were already customer. The device would already have been locked. We would customize it some more (injecting cryptographic keys, application, placing our labels on the device) and then send them to merchants. The merchants were never customers, they would get it on loan from us. This was the only way to do it as the device could not be re-used with other acquirer so it only functioned as long as the merchant had valid merchant account with us.

Transparent plastic?

They may find a way to hide things out of view. My thinking would be to x-ray it. Airports are pretty good at that.


> as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag

You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages [0] and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).

[0] https://www.techradar.com/news/networking/routers-storage/ph...

You can just buy counterfeit anti-tamper stickers but if there is a switch inside the unit that flips a bit in some sort of write-once memory, then that would require removal of an entire chip and replacing it with another that may not be 100% the same. You can have a chain of trust in the system where chips will only talk to each other if they all spit out the right hash. Bury the SPI/I2C lines you use for this trust check within the PCB so you can't access it without drilling the card and add a layer of anti-tamper traces that trigger another tamper event if disturbed. Now what was just a quick install has turned into a whole PCB rework job where you are having to swap all the chips with a virgin set, assuming you can't get your hardware in there prior to final assembly.

Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.

The device outer enclosure was tamper evident but the device itself was tamper proof HSM, basically. Any kind of intrusion (melting, dissolving, drilling, etc.) into a secure internal enclosure (separate processor, memory and battery) would cause internal battery to be disconnected from internal SRAM and basically the device would loose all cryptographic material and then self-destruct.

To give a bit of background, when you type your PIN on credit card terminal it is not the terminal application that is really getting the pin (well, except for special credit cards but that is really problem of the Bank that issued the card). The Visa/Mastercard mandate that the application don't have control over the PIN and that the PIN entry uses physically separate keyboard and display.

To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry and the PIN is transferred directly to the HSM where it is being encrypted before it is transferred to the application processor for the rest of processing.

> To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry

Perhaps this sounds too dull to ask, but what stops the terminal from just ... not separating the keyboard and display?

Certification. Worked in the same industry, and there were very strict both hardware and software requirements for POS software. Having gone trough credit-card audits, early EMV certification programs, and certification to place non-payment software next to payment software on such systems, I can tell you - it's no joke :)

> I can tell you - it's no joke :)

But still as a user I have no idea if I'm talking to a certified machine or not.

It's not your problem. For the transaction to take place it has to go to your bank and your bank trusts Visa/Mastercard to manage risks. You would be surprised to know most frauds are absorbed by banks. Nobody is interested in people fearing plastic because it causes people to increase very expensive debt plus additional interchange fee from every transaction.

As a user, fraud is not your problem. Security isn’t there for your benefit.

Certification [...] I can tell you - it's no joke :)

I'm not so sure. "And then initial supplier inserted hardware that thwarted all those pressing considerations" sounds a punchline to me. :(.

That, at least, is a problem that cab be solved by tightening security at supplier site.

The safeties are mainly to guard against the rest of the world. For example it prevents tampering in transit, or even in our own company - disgruntled employee can't do anything.

Or think for a second about the fact that we leave the device for, hopefully, entirety if its life at the client site. We had clients that were shady businesses like strip clubs that no other companies would touch with a stick.

Those VISA/MasterCard rules can't be universal because there's at least one bank issuing merchant terminals that run Android and take the PIN on the touchscreen:


Clover CEO here. Won't comment on a competing device but this may not work the way you think. In Clover's approach the touch controller input isn't reaching the Application Processor running Android when in PIN entry mode. You can do patent search if you're interested.

That would be in line with the requirements. You go through stringent certification with the software and hardware that has access to the actual PIN and then show that the application and application hardware never really has any access to it so that you can customize/update your software.

This is the easy part.

The hard part I remember was establishing secure communication between all components in the system (initializing HSMs, injecting keys). I remember helping designing the process and writing hundreds of documents describing various security-related procedures like how the HSM racks are inspected, how the keys to the racks are fetched from the safes, how there are multiple safes for multiple security officers, how the officers are prevented from ever having access to other safes, how fetching anything from safes requires logging and using tamper-evident containers, how the logs are inspected, and so on.

I have designed a special cryptographic protocol so that we could generate and inject keys to the devices in KIF (Key Injection Facility) and separately to our database (to establish communication with the terminal). Fun.

Agree the people and process side is very difficult to do well. Familiar with all those and more -- we have extremely good, dedicated employees who care deeply about doing those things right.

We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)

(If anyone reading is looking for a job in security engineering, we're hiring! https://www.clover.com/careers/engineering)

I have few more stories like the time when I closed the HSM rack door a bit too energetically and caused outage to entire company as we had to bring in third security officer to re-initialize it.

We also had special screens created for all cameras in the datacenter to block view on the HSM racks.

The biggest issue was, just before end-to-end test we figured out we forgot one of critical procedures (it was establishing authenticity of the HSM used) and we had to scramble to get new HSM and to re-establish all cryptographic material (so new storage keys, etc.)


What you're describing sounds like another backdoor of its own!

The general problem with most "industry security" approaches is that they simplistically attempt to wrestle ultimate Godmode-control for themselves, rather than working towards eliminating it.

Those are universal PCI council rules the fact that the terminal runs android doesn't mean anything it still needs to be compliant with the PIN entry device requirements as well as PTS and POI requirements depending on the exact nature of the device:



The requirement for the tamper proofings is literally the first requirement in the PED standard:

A1 Vendors must comply with all components of A1.


The PED uses tamper-detection and response mechanisms that cause the PED to become immediately inoperable and result in the automatic and immediate erasure of any secret information that may be stored in the PED, such that it becomes infeasible to recover the secret information. These mechanisms protect against physical penetration of the device by means of (but not limited to) drills, lasers, chemical solvents, opening covers, splitting the casing (seams), and using ventilation openings, and there is not any demonstrable way to disable or defeat the mechanism and insert a PIN-disclosing bug or gain access to secret information without requiring an attack potential of at least 25 per PED, exclusive of the IC card reader, for identification and initial exploitation as defined in Appendix B of the PCI POS PED DTRS

This only accept contact-less payment who doesn't require pincode.

It accepts contactless payments over the threshold where a PIN is required, and does in fact prompt for a PIN using an on-screen keypad.

PED/PTS devices have even stricter guidelines than contactless payments.

But this device doesn't need a way for the user to enter the pincode. So, all the sensitive part of the terminal is probably completely isolated from the android part.

I don't know this device internal and the PED/PTS exact requirement but it seems plausible for me.

You have something like a physical compartment who include the NFC and everything needed to process it like in a classical terminal. This compartment is highly secured as requested by the specification with just a very simple interface for the android part to send the amount to bill.

I've seen a lot of each-machine running on windows. Doesn't they work like this with the windows machine just managing the display buttons to select the amount and sending this information to the secure part who handle card interaction, pincode and delivery of the money ?

Yes this is a common design in fact most current solution segregated the POS and POI completely anything that handles the actual credit card whether it's C&P, Track2 or NFC is a closed black box with the required PED/PTS/POI and P2PE certifications the merchant never sees what's going on they only can talk to the thing in bill the next card X and get a confirmation of the transaction that's it they don't see any of the card data they don't even see any card holder data unless they collect it in a side channel e.g. a loyalty program.

Now none of these certifications or standards is bullet proof but people have a very skewed vision of the PCI certification process likely due to bias of only having interacting with the PCI-DSS requirements for merchants and low levels to boot meaning they didn't had to do anything but to fill the SAQ themselves and be on their way.

Worked in the payment industry for years. Visa/Mastercard do absolutely nothing to verify that companies are not storing Pin codes. The HSM is required for communication with them only.

That's not correct.

HSMs are required so that the company does not need to have PIN codes exposed anywhere. Not having PINs or full credit card data makes your life easier as there is nothing to steal from you in the first place.

If your company stored PIN codes it means you were in breach of the contract and it had to lie to the auditors to pass the certification.

It is correct. Incompetence abounds. You are correct about the HSM but it does not enforce anything except for the exchange between whoever and MC. You do realize that pin codes are entered into a UI and phone system as plain text right? There are PCI audits but they are a joke.

You realize when we talk pins we mean authorizing your credit card chip&pin transaction? Nothing to do with your phone pin or maybe som other pins.

The pin we are talking about is what is customized on your credit card (directly in its memory) or its equivalent in your bank's HSM for the sole purpose of performing CVM step negotiated by yor card and payment terminal.

"If someone was going to break the law, they would have to lie about it first."

Legit companies don't want the info and anyone that wants the info isn't doing anything legal with it.

That's not correct the QSA will validate that the device does not store PIN codes or the that the merchant does not store anything they are not allowed.

Devices that accept cards need to comply with PED/PTS security requirements including very strict physical security requirements which are validated by PCI council approved laboratories and firms.

You are not getting a device on the market or usable with any merchanet network without complying with this: https://www.pcisecuritystandards.org/documents/pos_ped_secur... and a few other standards.

About that, I can only say that Chinese android POSes do everything in software, for sure, without any hsm present.

The question is, how Chinese banks coax Visa into allowing them using them.

The POS and the PED/PTS isn't the same thing the POS can complete the transaction without touching the credit card in fact most of them do exactly that the only thing that it does is communicate with the PED/PTS to send the amount and get a confirmation/denial.

Simple greed? As in: "play ball, and you get access to the gigantic Chinese market"

Probably it just goes through as a card not present (CNP) transaction?

So you mean it first authenticates the pin, and initiates CNP after? Never thought of that as possible.

It's not CNP doesn't use the pin it uses the CVV2, you also can't use the chip and pin or track 2 swipe data for a CNP transaction.

I think the GP is confused on how a POS works, POS isn't a POI most of them don't touch the credit card they just talk to the reader, most readers today are P2PE closed loop solutions so the only thing the POS does is sends to the reader charge the next card $X the reader will then reply if the transaction went through or not and that's it.

The reader itself will talk to the acquiring bank or the payment provider in a point to point encrypted closed loop and the merchant would never see any credit card details.

Second this, after having to go through a service level 1 DSS review for a few years. Lower level reviews (3,etc) just require self validation.

SAQs don’t involve QSAs. They are also intended for merchants which are a rounding error also there is no SAQ for PA, PED, PTS etc. certifications only for merchant PCI-DSS.

You can totally fake your way through PCI audits. I know of a company that did it for years using a fake network and servers. Not sophisticated at all. Most auditors do not find all of the compliance violations. They have one person do it. It's all about money.

You can fake a lot of things so what? That’s not the point, also PCI DSS is pretty crappy but the hardware vendor, payment provider and P2PEE certifications are a completely different story good luck faking it.

Sure you can send fake devices to be certified and sell something completely different but the same can be said for any certification and if you get caught boy or boy...

Wrong. PIN codes are entered into a damn mobile app and passed through an API. Billions of times per day. You guys are clearly missing the card serciving aspect of the industry.

Please show me the device that transmits the pin of a chip and pin card to an API while not being compatible with PED and P2PEE requirements.

But in the US we just use signatures for card transactions instead of chip PIN :/

Do you honestly think any of that is not something a nation state actor could pull off though?

I never said a nation state couldn't attempt this but just that if you try really hard to lock down your hardware, it becomes very expensive and risky. You may be better off attacking the system in another way other than cracking it open and soldering your payload in there.

Payment systems are typically better defended than by just a sticker.

It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.

All of which are overcome by nation state actors if they want too.

Depends on the hardware and the anti-tamper measures. I've seen POS terminals where the pcb was completely encased in security plastic, where any attempted breach would wipe the internal security keys, which meant the hardware just became a useless. They're so sensitive that these things enter "tampered state" from time to time without any tampering. I developed software on these things - and bricked multiple devices by accident, even though this was 'development' hardware.

I have caused production outage during one of our reviews of the rack with the HSM. The procedure required opening the rack, inspecting and accounting for all devices and cables and then checking the status of the HSM itself. At the end of the procedure the rack had to be closed.

Visa/Mastercard required that there are two people present and that there are two people required to open the rack.

We had it modified so that it has second lock.

The additional lock did not fit perfectly. During the procedure when I tried to close the door the door snagged on the lock and then slammed shut. It wasn't a lot of force but it caused the HSM to loose its keys and required a lengthy procedure to get three security officers to initialize the HSM with components stored on their smart cards during which the whole system was unavailable.

The chinese government is probably not interested enough in credit card numbers to warrant involvement.

It would be foolish to show your hand for few pennies.

Might make a difference if those pennies can't be tracked back to the government.

Yup. This would be the modern equivalent of Air America and other schemes by the CIA to raise money to operate by involving themselves in illegal activity like the drug trade.

One would consider those activities to have taken place in "modern" times as well...

You're not suggesting the CIA is out of the drug business...are you?

Iran/Contra got the CIA into a lot of hot water with Congress. They had some pretty stringent restrictions put in place as a result.


People said the same thing about personal detail metadata as well. While theft would be foregone foolishness tracking transactions could have value to them. Granted it would probably be easier to snoop on the fiscal institutions.

What's the motive? If we assume a motive of theft, seems more reasonable that there's organized crime involved than a state actor.

> something as simple as an anti-tamper seal

To echo, it's actually quite trivial to bypass anti-tamper stickers with acetone and a needle.


Very interesting methods you used to detect the changes! Out of curiosity: was there a reason that taking an xray of the devices was not an option? Industrial/veterinary xray machines can often be had quite cheaply...

This feels cargo cultish. Products drop from the sky. One day they become poisonous. You have no idea how to reproduce them locally. So you come up with hacks to make then less dangerous.

We really need to get back into manufacturing if this is our brave new world.

The better question is how did we stop manufacturing.

Get an x-ray machine? They are surprisingly cheap pieces of hardware if you are willing to deal with a small area, low penetration image. Low penetration means no lead, which makes for something that's about as cumbersome as a large bar fridge.

It's mentioned in the article that X-Ray didn't help much: 'Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment'

This comment is specific to the parent talking about their experiences producing credit card terminals that ended up with PCBs implanted in them. Here it is appropriate.

And as soon as they know you are using X-Ray to test ( I would be surprised if there were no insider information leaking on the OP's end ), they would develop a way so that somehow the X-Ray images looks identical. May be a heat spreader like metal as cover up.

Can't the image be diffed with one of a trusted system?

> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.

You only need to measure three angular momentums, all other can be calculated. See https://en.wikipedia.org/wiki/Moment_of_inertia#Motion_in_sp...

"This shows that the inertia matrix can be used to calculate the moment of inertia of a body around any specified rotation axis in the body."

On the attacker side, they only need to make sure three angular momentums match.

Maybe they're also measuring centrifugal force for rotations with axes not intersecting the centre of mass.

For the folks commenting below that we should bring the manufacturing back to the US, why wouldn't the bad guys just start bribing American workers to insert the attack hardware into devices made here?

It's not like Americans are somehow above being bribed.

The bad actors could be brought to trial in a United States court, which is a level of deterrent not included in offshore manufacturing. If a US manufacturer was found selling tampered chips then the company itself could be held liable. This would create a general pressure to increase onsight security.

There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.

Americans can be bribed to do anything. However, you don't really need to bribe or ask someone to betray their country and fellow citizens when it's done on Chinese soil...

It's possible, but it's much, much, much easier for an American company to hold another American company accountable when something like this happens. Instead of having to go through all those hoops, you sue the pants off the manufacturer.


Nationalistic slurs will get you banned here. Please don't stoop to that again.


I think people here are just suggesting that the folks who believe producing things in America would avoid a culture of abject amorality and greed seem naive, at best. Even ignoring the current obvious examples, Americans frequently engage in profitable criminal behavior: https://www.businessinsider.com/mcdonalds-monopoly-game-rigg...

This reminds me of a story about, IIRC, Soviet intelligence personnel determining that a photocopy machine at their consulate in the USA was bugged by the CIA by measuring it's weight and comparing that value to the standard value published by Xerox.

Modern times, but same old methods of "debugging"

The angular momentum stuff is innovative though.

Now I’m picturing a 300lbs photocopier being spun on three axes. How much tape would it take to keep doors and parts from flying off?

is measuring the moment of inertia not standard? I think any mechanical engineer would suggest it for that kind of problem (disclaimer; I've suggested it before for the same problem)

A truly sophisticated attacker would leave some easily detectable bad eggs in the stream after countering a countermeasure like the weighing.

Why don't you guys consider to expose this by suing the bad manufacturer? I believe this could help other truely honest manufacturers both in and outside China to beat the wrong doers.

The Chinese operation would simply shut down and reopen under a different name. And the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.

But I don't think it's that good to keep playing the rat-cat game.

> the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.

Letting the bad guys keep doing what they do while putting it's own customers under shadow, to me, it sounds like the credit card companies are helping the bad guys.

As a Chinese myself and a credit card user, I'm more worried than you guys do, because we are more likely to receive that kind of treatment (compromised credit card and computer chips etc). I really hope somebody can teach those bad doers a unforgettable lesson.

I believe you are talking about "Rotational Moment of Inertia" about various axis, instead of "Angular Momentum".

Angular Momentum of a body at rest is zero.

Could this be solved by a clause in the contract that specified if any randomly sampled devices came tampered with, the manufacturer did not get paid?

Ah, yes, and then we would voluntarily go out of business. And the supplier will just have another customer that would not be so principled.

Huh? This would be between you and the Chinese manufacturer. If you found s tampered unit, you don’t pay them. How should that make you go out of business?

They need the units from somewhere, with a deadline determined by their own runway, and presumably only some of the Chinese ones were compromised. Sure, they could avoid paying, but then they have to spend months getting the production run going somewhere else .. which might also be compromised.

I'm sure the Chinese government would be willing to keep afloat companies that allow them to tamper with.

I do not for one minute believe, that china, whose production floors are spotless, watched like a hawk and every product tested, did not know this was being added. They would have been able to tell themselves by weight. Yes, product weight is used for shipping manifests too.

This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.

At what point does it get cheaper to buy from sources that are more expensive but more trustworthy?

>measuring angular momentum on a special fixture

Can you elaborate on this concept a bit? I'm not familiar with this term and the sources I looked up were pretty physics-y and out of my depth.

This sounds like a very interesting and creative solution. Good lateral thinking on your part! :)

What good is angular momentum when the producer can have fluctuations in its supply chain? Yes you can see when devices are not the same, but what if that happens all the time, legitimately?

Was the secret service contacted? Card skimmers are a big no no. Family friend works for Dept of weights and measures and finds skimmers all the time on gas pumps. Scary stuff.

Gosh, I would love to read everything about this. I know there are some videos about anti-tampering card readers on youtube, but not on the feedback race between hackers and security.

Maybe a setup to measure inertia tensor and center of mass (in that setup's axes) will be easier and I think it's what your call "measuring all angular momentums".

Thanks for sharing this story, and I hope you aren't put off by the huge thread of people second-guessing your competence. We need more of your kind of story.

x-ray would have been easier, as others have said.

but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.

in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).

Why do you use those suppliers?

Maybe time to buy your financial hardware from somewhere not china?

I know it's nice to blame China for everything... but it's not really the root of the problem here, supply chain management and control is.

China is 100% to blame here.


So you had hard evidence that your manufacturer was vandalising your property? Couldn't you drop them?

The main lesson I learned from my Dad’s employer (twice) was always have two vendors. You can play them off of each other. When I did contract work I saw how powerful getting out or putting for away the checkbook can be. Large vendors ignore you if the checkbook isn’t moving.

I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.

Did you look into Terahertz scanning?

That's kinda amazing

Sorry, but this doesn't sound as true or there is huge mistakes done in choosing supply chain for such sensitive matter.

How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?

How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).

How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.

What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?

Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.

While it may sound sensational this was more of an operational issue, really. We were told by Visa and Mastercard that it is not even a question if we are going to be targeted. If you work in payment card industry you are constantly being attacked and the only way is to make it part of the process to deal with those things. Our network was hacked but what was important was tight, almost mathematical processes around protecting very specific material like credit card data and PINs.

For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.

It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.

Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking about malicious implants. Just google for it, it’s massive

It sounds like they are just ordering stock products from Amazon.

Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking of malicious implants

This is fairly analogous to my comment here: https://news.ycombinator.com/item?id=18138847

Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.

Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab. Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.


I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact