I have worked in card payment industry. We would be getting products from China with added boards to beam credit card information. This wasn't state-sponsored attack. Devices were modified while on production line (most likely by bribed employees) as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag.
Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).
They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.
We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
First, wow this is both incredible and crazy! Both the China-side hacks and your side's anti-hack. Mind. Blown.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
This wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China.
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.
> Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?
America has fabs, both old and leading edge, but ask industry giants like Gemalto to even bother to manufacture chips anywhere outside of Taiwan, assemble the final product outside of China.
They will never do that, because they look for the cheapest solution.
The bigger the company, the less it cares about things other than cost. This is why Mediatek and Broadcom can usurp the market of network SoCs, while making products with atrociously bad support. I personally dealt with both, and say that they wholely match their popular culture image.
I don't know how it is with USA, but for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres, for as long as they get them made inside the country.
Gemalto manufactures and assembles many, many products in Europe. The entire European security cluster won't use anything else. I know because I played a part in smart card development there. And all of that was from Europe. Wafers, chips, holograms, mag stripes...
Not exactly, PCB assembly is super cheap everywhere thanks to propagation of chipshooters, what makes the cost go up is logistics - what do you do after you populate the board for your part? Ship it across the world, or to another factory behind the corner?
That largely depends on the size of your board and the total number you want to ship. As soon as you reach full truck loads or full container loads that additional shipping cost is marginal on a board level.
You are right in a certain way. What increases with distance is lead time and as result inventory. Both of which are off set by volume. Cross-border adds customs issues. Complexity is not that much of a problem nowadays, maybe it never was. You are right that all of this oblyakes sense with large numbers, small scale production is better done locally as a rule of thumb.
> for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres
Well, if your chips are bigger and slower, you will need more chips and mounts/packaging to place them. If you need more chips then the weight of missile/plane/tank will be increased and available space decreased.
So at the end, the 'uncaring' military will receive a weapon which is worse than competition.
I think people should know how stark the differences for assembly in the US are vs outside of it. Something that costs, at low prototype volumes mind you, $20 in China for a dozen boards or so, would run hundreds of dollars in the US and still take the same amount of time. As it scales up, the ratio might improve, but these aren't like 10%-20% differences.
Nationalistic flamewar is not welcome here; neither are political or ideological flamewar. We ban accounts that use HN primarily for those things. Please read https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here.
at first I had the same thought. but i have to question how securely the same manufacturing could be done in a US plant.
the US employee base has its fair share of desperate, ethically challenged individuals. and plenty of incentives to make a quick buck could be offered here too. idk.
OTOH, given US law enforcement's low efficiency what are the chances of being caught? and what if it's merely corporate espionage?
finally, the US is an open society with strong personal freedom guarantees builtin. what if the perpetrator has ties to a foreign country and simply leaves the US after they've installed the vulnerabilities?
You are underestimating the FUN of playing anti-anti-^N-hacks. I have had the privilege to be paid to so anti-anti-^N-hacking on a firewall thingy in the past and it was a challenge and a joy!
Honestly if it weren't programming I feel like this is a movie-worthy story. To me it sounds so thrilling like a spy movie plot but am I just imagining that or was it actually this crazy / cool / integral to way bigger moving parts / things like I'm assuming? Regardless kudos. Your story definitely started my day on a happy note, thanks!
We had MasterCard end-to-end test auditor on site. This is the first time ever you get to do a transaction with real transaction system with real credit card.
Due to requirements we opted to have the only large meeting room to have outside our secure zone. This created an issue as we had no network access from there and in the end we decided to use slow GPRS terminal for the test.
The end-to-end test starts with offline transactions which by their very nature are quite fast (it is negotiated between terminal and card).
But then we went to online transaction and it finished instantly too.
The auditor, bewildered, proclaimed the test failed as he assumed it was incorrectly processed offline instead of going online. But then I pointed out to the printout to show ARQC (basically says it was certified online).
Now, the real discussion started. The terminal was very slow taking quite few seconds to establish GPRS and then even more for the SSL handshake so the auditor said it was not possible to make it work.
How it worked was that I have completely gutted OpenSSL and had entire cryptographic state stored locally (safely, using internal HSM) so the SSL session could be optimistically re-established without another handshake even after TCP connection was closed. The first message the terminal sends is already encrypted transaction message, there is no SSL handshake. I wrote an application to terminate the connection in our data center so that it stored the states of each connection in the database. The entire handshake was only done if the first message could not be decrypted successfully.
The operating system was single-threaded with no multitasking of any kind. This meant that all applications on this device did their operations sequentially. Send network message, print something, display something, etc.
I wrote a cooperative multitasking functionality into the application (using coroutines) so that it could work on multiple tasks at the same time (like talking to network and printing).
I then have segregated all data on the printouts so that it can start printing without having to already have response from network. Hopefully if everything went right, the response would come before it even came to that place on the printout effectively looking as if it was done in zero time.
FWIW, the described technique (or something roughly equivalent) is now standardized as 0-rtt early data in TLS 1.3. (you still need 1-rtt for TCP, unless you can combine this with tcp fast open, or run TLs over UDP)
But am I wrong to have my hackles raised by a) the roll-your-own security nature of this, b) the reliance on a single developer's single stack implementation as what guarantees the integrity of the system? It seems like there are a lot of assumptions baked in.
I, too, would love to see a more detailed write-up--if there's a big idea here (almost a unikernel thought), it deserves to be shared and tried by fire.
When I worked in telecom (a while ago) the manufacturing was shifted from China to Thailand/Other SE Asia due to this. The Thai companies weren't as efficient, but were much more open and honest when problems would arise., plus they didn't blatantly steal tech
See, the article showed that even largest companies are not completely immune to the problem. This was decade ago and payment card industry, not exactly national security matters.
The PLA can lean on factory managers very effectively but they're not going to be interested in small time stuff like credit card numbers. The sort of sophisticated criminal gang doing something like this will have fewer coercive tools at its disposal and I'd imagine would target lower level employees with bribes.
All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia and these are quarantined immediately after they return. They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).
The more critical the field, the more you have to treat those devices as untrusted before attaching them to your trusted zone.
> They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).
Given how sophisticated these attacks can be, I'd think they'd issue disposable equipment to be destroyed on return, like a cheap netbook or something. I don't see how you could trust an individual viewing a simple X-ray scan to detect some extra microchip the size of a signal conditioning coupler.
Procedures change, as attacks get more sophisticated the next step could be disposable devices. But an attack like the one described in this article won't be mitigated by having a disposable device. On the other hand having your laptop "hijacked" while on a business trip will most likely involve some extra PCB or components that are a little more obvious that something that's "built in".
Then again many companies or public institutions would find it hard to justify shredding each week maybe tens of laptops and phones that still have to be good enough to work on. Basically they still have to be a "standard issue" device with your company's software stack, config, etc.
I'm sure someone can find a good compromise between security and wastefulness.
The obvious solution would be to earmark some number of dedicated "china laptops" that will never be trusted with anything important, and get reused everytime someone has to visit China. If they get backdoored... who cares? They can only spy on things that were getting spyed on anyway due to being in China.
Does the security team have any motivation to reduce waste? Unless you're finding hacks in your devices already I see no reason to think they aren't just making you jump through hoops because it's funny.
The post-trip inspection is not so that the device can be reused, it's so you can (try to) find out if it was compromised. A $3000 laptop is not a significant cost compared to the airfare, hotel bills, etc.
But it's useful to know when/if you're being targeted.
One of the companies named in thr Bloomberg article does. They just deatroy your laptop if it was in the hands of customs without your supervision for any length. US customs explicitly included, which is kind of wierd if you ask me.
That’s perfectly expected. It’s not a stretch of the imagination to think border checks are abused for industrial espionage. If it gives your country a major advantge nothing is off limits these days.
My employer - doing lots of interesting engineering in among others the marine and aerospace sectors - certainly do that; we're based in Europe.
Basically, any device brought to the US and a number of other countries are issued for one-time use; if they leave our custody for even an instant, they are to be scrapped.
> All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia
From what I understand, Boeing does this when employees visit France.
They have had problems with men in nice suits going through laptops stored hotel safes.
Sure, we were discussing someone saying their devices came with extra PCBs inside (it's a bit hard to follow but scroll up to the original comment, currently first on the page).
> We would be getting products from China with added boards to beam credit card information.
>> Trying to guess the contents of a box that you cannot open sounds a bit like madness.
>>> Use X-ray? or whatever can penetrate the exterior shell
2 different types of attacks, 2 different types of responses.
Yes, irrespective of country where its manufactured, if there are compliance requirements around an un-openable box, then some process becomes required.
But I think the GP's question is: "Whether it would be cheaper" - in the sense whether such an expensive QA process could have been averted by having a more trustworthy partner. One whom you're not on a race hack after hack.
The point is that if the devices are sensitive with compliance requirements then you must be able to verify them irrespective of who you hired to manufacture them.
You cannot just trust the word of a contractor on this because it's your ass on the line.
The point is that the process was to assure the device wasn't tampered AFTER shipped from manufacturer. Nobody thought it could already have been modified so early in the process. This is the eternal cat and mouse game. When I started in IT in 90s it was assumed that company network was quite safe and you didn't always need passwords, maybe for critical resources only.
I would think that, logically, and as illustrated, "the device wasn't tampered AFTER shipped from manufacturer" means after YOU have shipped it to customers. The anti-tampering system is to prevent modifications in the field.
The manufacturer shipped the device to us from China. We were already customer. The device would already have been locked. We would customize it some more (injecting cryptographic keys, application, placing our labels on the device) and then send them to merchants. The merchants were never customers, they would get it on loan from us. This was the only way to do it as the device could not be re-used with other acquirer so it only functioned as long as the merchant had valid merchant account with us.
> as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag
You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages [0] and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).
You can just buy counterfeit anti-tamper stickers but if there is a switch inside the unit that flips a bit in some sort of write-once memory, then that would require removal of an entire chip and replacing it with another that may not be 100% the same. You can have a chain of trust in the system where chips will only talk to each other if they all spit out the right hash. Bury the SPI/I2C lines you use for this trust check within the PCB so you can't access it without drilling the card and add a layer of anti-tamper traces that trigger another tamper event if disturbed. Now what was just a quick install has turned into a whole PCB rework job where you are having to swap all the chips with a virgin set, assuming you can't get your hardware in there prior to final assembly.
Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.
The device outer enclosure was tamper evident but the device itself was tamper proof HSM, basically. Any kind of intrusion (melting, dissolving, drilling, etc.) into a secure internal enclosure (separate processor, memory and battery) would cause internal battery to be disconnected from internal SRAM and basically the device would loose all cryptographic material and then self-destruct.
To give a bit of background, when you type your PIN on credit card terminal it is not the terminal application that is really getting the pin (well, except for special credit cards but that is really problem of the Bank that issued the card). The Visa/Mastercard mandate that the application don't have control over the PIN and that the PIN entry uses physically separate keyboard and display.
To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry and the PIN is transferred directly to the HSM where it is being encrypted before it is transferred to the application processor for the rest of processing.
Certification. Worked in the same industry, and there were very strict both hardware and software requirements for POS software. Having gone trough credit-card audits, early EMV certification programs, and certification to place non-payment software next to payment software on such systems, I can tell you - it's no joke :)
It's not your problem. For the transaction to take place it has to go to your bank and your bank trusts Visa/Mastercard to manage risks. You would be surprised to know most frauds are absorbed by banks. Nobody is interested in people fearing plastic because it causes people to increase very expensive debt plus additional interchange fee from every transaction.
That, at least, is a problem that cab be solved by tightening security at supplier site.
The safeties are mainly to guard against the rest of the world. For example it prevents tampering in transit, or even in our own company - disgruntled employee can't do anything.
Or think for a second about the fact that we leave the device for, hopefully, entirety if its life at the client site. We had clients that were shady businesses like strip clubs that no other companies would touch with a stick.
Those VISA/MasterCard rules can't be universal because there's at least one bank issuing merchant terminals that run Android and take the PIN on the touchscreen:
Clover CEO here. Won't comment on a competing device but this may not work the way you think. In Clover's approach the touch controller input isn't reaching the Application Processor running Android when in PIN entry mode. You can do patent search if you're interested.
That would be in line with the requirements. You go through stringent certification with the software and hardware that has access to the actual PIN and then show that the application and application hardware never really has any access to it so that you can customize/update your software.
This is the easy part.
The hard part I remember was establishing secure communication between all components in the system (initializing HSMs, injecting keys). I remember helping designing the process and writing hundreds of documents describing various security-related procedures like how the HSM racks are inspected, how the keys to the racks are fetched from the safes, how there are multiple safes for multiple security officers, how the officers are prevented from ever having access to other safes, how fetching anything from safes requires logging and using tamper-evident containers, how the logs are inspected, and so on.
I have designed a special cryptographic protocol so that we could generate and inject keys to the devices in KIF (Key Injection Facility) and separately to our database (to establish communication with the terminal). Fun.
Agree the people and process side is very difficult to do well. Familiar with all those and more -- we have extremely good, dedicated employees who care deeply about doing those things right.
We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)
I have few more stories like the time when I closed the HSM rack door a bit too energetically and caused outage to entire company as we had to bring in third security officer to re-initialize it.
We also had special screens created for all cameras in the datacenter to block view on the HSM racks.
The biggest issue was, just before end-to-end test we figured out we forgot one of critical procedures (it was establishing authenticity of the HSM used) and we had to scramble to get new HSM and to re-establish all cryptographic material (so new storage keys, etc.)
What you're describing sounds like another backdoor of its own!
The general problem with most "industry security" approaches is that they simplistically attempt to wrestle ultimate Godmode-control for themselves, rather than working towards eliminating it.
Those are universal PCI council rules the fact that the terminal runs android doesn't mean anything it still needs to be compliant with the PIN entry device requirements as well as PTS and POI requirements depending on the exact nature of the device:
The requirement for the tamper proofings is literally the first requirement in the PED standard:
A1 Vendors must comply with all components of A1.
A1.1
The PED uses tamper-detection and response mechanisms that
cause the PED to become immediately inoperable and result in the
automatic and immediate erasure of any secret information that may
be stored in the PED, such that it becomes infeasible to recover the
secret information. These mechanisms protect against physical
penetration of the device by means of (but not limited to) drills, lasers,
chemical solvents, opening covers, splitting the casing (seams), and
using ventilation openings, and there is not any demonstrable way to
disable or defeat the mechanism and insert a PIN-disclosing bug or
gain access to secret information without requiring an attack potential
of at least 25 per PED, exclusive of the IC card reader, for
identification and initial exploitation as defined in Appendix B of the
PCI POS PED DTRS
But this device doesn't need a way for the user to enter the pincode. So, all the sensitive part of the terminal is probably completely isolated from the android part.
I don't know this device internal and the PED/PTS exact requirement but it seems plausible for me.
You have something like a physical compartment who include the NFC and everything needed to process it like in a classical terminal. This compartment is highly secured as requested by the specification with just a very simple interface for the android part to send the amount to bill.
I've seen a lot of each-machine running on windows. Doesn't they work like this with the windows machine just managing the display buttons to select the amount and sending this information to the secure part who handle card interaction, pincode and delivery of the money ?
Yes this is a common design in fact most current solution segregated the POS and POI completely anything that handles the actual credit card whether it's C&P, Track2 or NFC is a closed black box with the required PED/PTS/POI and P2PE certifications the merchant never sees what's going on they only can talk to the thing in bill the next card X and get a confirmation of the transaction that's it they don't see any of the card data they don't even see any card holder data unless they collect it in a side channel e.g. a loyalty program.
Now none of these certifications or standards is bullet proof but people have a very skewed vision of the PCI certification process likely due to bias of only having interacting with the PCI-DSS requirements for merchants and low levels to boot meaning they didn't had to do anything but to fill the SAQ themselves and be on their way.
Worked in the payment industry for years. Visa/Mastercard do absolutely nothing to verify that companies are not storing Pin codes. The HSM is required for communication with them only.
HSMs are required so that the company does not need to have PIN codes exposed anywhere. Not having PINs or full credit card data makes your life easier as there is nothing to steal from you in the first place.
If your company stored PIN codes it means you were in breach of the contract and it had to lie to the auditors to pass the certification.
It is correct. Incompetence abounds. You are correct about the HSM but it does not enforce anything except for the exchange between whoever and MC. You do realize that pin codes are entered into a UI and phone system as plain text right? There are PCI audits but they are a joke.
You realize when we talk pins we mean authorizing your credit card chip&pin transaction? Nothing to do with your phone pin or maybe som other pins.
The pin we are talking about is what is customized on your credit card (directly in its memory) or its equivalent in your bank's HSM for the sole purpose of performing CVM step negotiated by yor card and payment terminal.
That's not correct the QSA will validate that the device does not store PIN codes or the that the merchant does not store anything they are not allowed.
Devices that accept cards need to comply with PED/PTS security requirements including very strict physical security requirements which are validated by PCI council approved laboratories and firms.
The POS and the PED/PTS isn't the same thing the POS can complete the transaction without touching the credit card in fact most of them do exactly that the only thing that it does is communicate with the PED/PTS to send the amount and get a confirmation/denial.
It's not CNP doesn't use the pin it uses the CVV2, you also can't use the chip and pin or track 2 swipe data for a CNP transaction.
I think the GP is confused on how a POS works, POS isn't a POI most of them don't touch the credit card they just talk to the reader, most readers today are P2PE closed loop solutions so the only thing the POS does is sends to the reader charge the next card $X the reader will then reply if the transaction went through or not and that's it.
The reader itself will talk to the acquiring bank or the payment provider in a point to point encrypted closed loop and the merchant would never see any credit card details.
SAQs don’t involve QSAs.
They are also intended for merchants which are a rounding error also there is no SAQ for PA, PED, PTS etc. certifications only for merchant PCI-DSS.
You can totally fake your way through PCI audits. I know of a company that did it for years using a fake network and servers. Not sophisticated at all. Most auditors do not find all of the compliance violations. They have one person do it. It's all about money.
You can fake a lot of things so what? That’s not the point, also PCI DSS is pretty crappy but the hardware vendor, payment provider and P2PEE certifications are a completely different story good luck faking it.
Sure you can send fake devices to be certified and sell something completely different but the same can be said for any certification and if you get caught boy or boy...
Wrong. PIN codes are entered into a damn mobile app and passed through an API. Billions of times per day. You guys are clearly missing the card serciving aspect of the industry.
I never said a nation state couldn't attempt this but just that if you try really hard to lock down your hardware, it becomes very expensive and risky. You may be better off attacking the system in another way other than cracking it open and soldering your payload in there.
Payment systems are typically better defended than by just a sticker.
It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.
Depends on the hardware and the anti-tamper measures. I've seen POS terminals where the pcb was completely encased in security plastic, where any attempted breach would wipe the internal security keys, which meant the hardware just became a useless. They're so sensitive that these things enter "tampered state" from time to time without any tampering. I developed software on these things - and bricked multiple devices by accident, even though this was 'development' hardware.
I have caused production outage during one of our reviews of the rack with the HSM. The procedure required opening the rack, inspecting and accounting for all devices and cables and then checking the status of the HSM itself. At the end of the procedure the rack had to be closed.
Visa/Mastercard required that there are two people present and that there are two people required to open the rack.
We had it modified so that it has second lock.
The additional lock did not fit perfectly. During the procedure when I tried to close the door the door snagged on the lock and then slammed shut. It wasn't a lot of force but it caused the HSM to loose its keys and required a lengthy procedure to get three security officers to initialize the HSM with components stored on their smart cards during which the whole system was unavailable.
Yup. This would be the modern equivalent of Air America and other schemes by the CIA to raise money to operate by involving themselves in illegal activity like the drug trade.
People said the same thing about personal detail metadata as well. While theft would be foregone foolishness tracking transactions could have value to them. Granted it would probably be easier to snoop on the fiscal institutions.
Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?
No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."
I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.
There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company.
The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).
If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.
All employee actions are company actions. You partnered with a company that can’t control what it’s employees do? No internal audits to make sure their reputation wasn’t being tarnished by a few employees?! Your loss.
I understand the indignation etc etc. And the suggestion to not use these kind of companies anymore.
And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it.
One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets.
The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.)
If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
How much do you think you would have to pay them to make it worth their while to not use cheap components (so as not to risk losing you as a customer)? It’s amazing how people never consider using economics.
Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?
As somebody else noted: it doesn't have to be the company that does it, but a lower level employee who cuts a deal. So paying the company more wouldn't solve the issue completely.
All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.
And again, you should make it the company's problem if they can't control their own employees. They can learn to put some controls in place once it starts hurting their bottom line.
> All of these things are extremely cost sensitive.
Say their profit margin is 1%. If you offer to pay them a 1% higher price, you're offering to nearly double their profits on whatever they sell to you.
Admittedly, that only works if you are a sufficiently large portion of their business. If you're a tiny percentage, then it may take a lot more to motivate them if their other customers aren't willing to pay more.
If the solution to the problem involves line checks, independent audits, personnel checks, increased pay, etc., thenyes, throwing money at the problem is a large part of the solution.
You have to pay them as much as it would cost to manufacture in the Western country doing the out-sourcing...so unless the cost of these extra quality checks outweighs the cost of on-shoring you might as well measure angular momentum.
IMO this is not indignation, it's supplier sourcing 101.
> If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
I will definitely agree that holding suppliers to standards regarding consistent output, unadulterated products, and conducting audits all make production much more expensive.
But if you're a device manufacturer, these sourcing controls are key to shipping quality products.
But make no mistake, this yet another globalized race to the bottom, but one that hits capital (or the end consumer), rather than labor or the environment.
>The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty
Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).
I used to work on the server-side stuff for telecom devices. We designed hardware that went into customer homes, but only downloaded the (encrypted) firmware upon home activation as otherwise the Chinese manufacturers would have ripped us off and sold them to telecom companies without our cut.
So, realising they could copy our hardware, but didn't have our software, they responded by trying to hack my servers, multiple times, from the same IP they sent manufacturing data from. A quiet word with their management would stop it, and it'd start again a couple of days later.
These people have no shame, and if we are going to go for lowest cost at all times it is what manufacturers should expect to happen.
OMG, that is so so so so brilliant! I'm going to go tell my boss right now that the suppliers better suite up, otherwise we are going right to their more expensive, less experienced vendor and will delay our product launch for a year....and likely still suffer the same problem. Those Chinese vendors better shape up or we're going to really teach them a lesson by driving ourself out of business right quick!
Just imagine for a second if someone was advocating on HN for the use of a web framework that has knowingly allowed itself to be compromised to steal passwords because it was cheaper to use.
How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.
Leaving an abusive relationship tends to be scary, especially if the victim is convinced that the abuser is the least bad. Doesn't mean the thought of ending that relationship should be dismissed out of hand.
You can be as sarcastic as you want, but these stories are absolutely indicative of a much larger problem. It might be beyond the capacity of one company to fix, but in the aggregate they represent a serious political and economic threat and need to be dealt with one way or another. Preferably by literally anyone other than Donald Trump.
Speaking as someone who is _not_ a citizen of the USA and is well aware of his issues, he seems to be first US President who actually appears to have the guts to do something about it. I doubt an establishment politician would ever have imposed tariffs on 200 billion dollars of China imports.
If he stuck to just China I would agree. But by trying to change up numerous different agreements, he lost the goodwill needed to have a collective crackdown on China. Even if he is the first to be willing to pick these sort of battles in a long while, he has a new issue of picking too many battles. Seems like a Goldilocks problem.
The tariffs aren’t the problem, it’s the stupidity of going it alone and alienating allies that could have turned the thumbscrews with you on China to get them to really capitulate.
> some employees were doing it with no involvement of the company
Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.
>There were other considerations like the fact we were actually buying it from large reputable company and what happened was that some employees were doing it with no involvement of the company.
It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way.
Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?
the indignation was not that. you are still missing the point.
after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.
Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.
> Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.
I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened.
However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.
PCI DSS allows for "Mitigating Controls" if you need to deviate from specified requirements, provided it is well documented and is equal to or greater in security. Doing teardowns to review circumspect hardware, and applying one's own tamper protection deal (and with accompanying documentation and tracking/logged information) would very likely be sufficient to maintain complaince.
I am reading it as "we don't any any option outside China and we have to live with this". I guess even after adding cost of doing additional of products you buy from China its less than what you would pay to buy it from other place.
Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.
>If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.
> they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them.
I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs.
And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify.
You could also be mandating how many boards are allowed or that the SNs go on early in the build process.
This is just how low trust societies function for hundreds if not thousands of years. This is also why families tend to be much stronger in these places. The American idea of a strong family is different from what I'm describing, since it just centers on immediate family. My definition of strong family bonds, implies both closeness and dependency not just on your immediate family, but even on your distant cousins on both sides. It's close if not the same to how Mediterranean and Hispanic cultures view families. I guess you can call it a clan mentality. I'm now wary of countries where families are really strong ie clans. It's not always the case, but it tends to mean the rule of law is weak and corruption is crazy.
When you can't even trust that food is real, and when you have to bribe even low level provincial government employees (can you imagine needing to bribe DMW workers or even police?) - what else is an average person to do except to treat it as the norm in order to survive? Unless you're an elite, the only other option is to leave, and not everyone has that choice. Very little is considered wrong over there, aside from criticizing the powers that be; as long as it helps your clan. It doesn't help when Western companies and governments look the other way, as we see in past news stories and even the comments here.
Not much to add, but want to note that you're spot on. I've encountered this SO many times over the years and this is always the explanation I'm given. At first it just made me indignant, too, and to some degree I guess it still does, but now I just try to let it go.
It's not rewarding bad behavior when you take it into account in the deal. Moralization is useless when you do business with different cultures. When you notice something what would be out of ordinary in your culture, you estimate the cost and find out how locals solve it. Different cultures have different kinks and people living there know to adjust to them. It can be fun to figure out how other cultures function.
From a naive Finnish perspective you could see American business culture as dishonest. People lie all the time and promise things that are false and you need massive amount of legalese or they fuck you over. When they say something, you can't trust it as much as you can trust a Finn (or Swede). American way is to lie to your face in pre-negotiation but then be truthful to the legally binding agreement. Germans or Belgians are more truthful beforehand and they stick to deals better without legal enforcement looming over them, but require more detailed written deals than Finns. Who is right? Should we punish Americans for their behavior or adopt their culture?
Of course not. You must understand that that the level of acceptable exaggeration, overpromise and bullshit and trickery is strictly culture dependent and relative. Someone is dishonest if they "lie this amount above average" in their cultural context. After you make an adjustment, you know how to get the truth.
Chinese manufacturers have the capability to stick to standards, take responsibility and deliver high quality stuff. It's just more dependent on personal relationship. Just placing an order has less trust as default. Either you develop "quanxi" with your suppliers or work trough some third party that understands both worlds. Third party understands what you need and knows how to get it from the Chinese suppliers because they have a relationship.
Chinese pay the cost that comes with their business culture. They can't network as fast as you can in open western cultures with more default trust between strangers.
Altering components to something cheaper is common practice for every company that is willing to exploit the fuck out of their customers. Look at the food industry. In EU there is control, in US thare are sanctions. But it is up to the governments because companies are to opportunistic to change. As long you can (re)sell the junk you get, you're fine, if you can't you upgrade "QA".
Both the EU and US have internal controls on food quality and sanctions.
Contrary to the impression your post gives, the EU has much more "sanctions" against food (esp. imports) than the US does, and is generally taxing much heavier and restricting many more things. Importing essentially any milk product into the EU is only possible on an exception basis. Trump has a LOT more work before extra sanctions in the US will become comparable to the EU ones.
What is often not said is that the EU not only has sanctions on its borders against food products, it actually has internal sanctions and regulations that are purely political in nature.
For example "champagne" in the EU can only be made within a particular region in France. Which leads to funny, in a sad way, consequences:
For your other part of your comment, I am unwilling to give a real answer. Obviously the US checks it's food quality. It's made different choices than the EU, famously the US allows hormone "enhanced" meat, but it does check food quality and safety. There are heavy penalties for breaking them.
Champagne is literally the name of the area where the product comes from. It makes perfect sense that anything that doesn't come from the region Champagne, isn't champagne... Call it sparkling wine, it's fine, but not champagne. And it applies to most local products. Camembert? Comes from a specific town, which is called Camembert. Beaujolais? The name of the province. And the list goes on. It doesn't sound "absolutely ridiculous" to me.
Did you look at the map? It's nearly all things with a place name in the product name. For example, "Welsh Beef". Why is it reasonable for a consumer to buy Welsh Beef and for it to possibly have come from somewhere other than Wales?
It seems to me, more like generalising the concept of trademarks to place names rather than just company names and brand name.
Not all cultures are great. If you are claiming “all cultures are great”, then perhaps “great” has lost any useful meaning.
Discrimination, as the ability to discriminate features to identify useful pattens, is not *-ist; it’s a critical skill that is being sullied by some neo-intellectual BS that passes for “equality” or something.
The ability to identify patterns of behaviour prevalent to a class of suppliers and avoid them, rewarding other supplies — even from the same area of the world — who are better; even “great” perhaps, is a valuable skill...
But perhaps I’m wrong. They’re all great. Carry on.
Give one example of a non-ethnic Chinese person who is considered to be culturally Chinese.
It's the norm in the west to not conflate culture with race. But to think that this is the norm in the rest of the world is to project your value system on others.
I had the same initial gut/emotional reaction, but after a re-read, not _sure_ if it's justified.
If we define "a culture" as set of sociological customs, value systems, expected behaviours and stimuli/response criteria;
and if we define "culture greatness" as such sociological customs, value systems, and behaviours that lead to specified and asserted goals;
than one can reasonably compare and contrast cultures in meeting (or even striving) to these goals, and thus discuss their greatness.
Only to the extent that the relationship between race and culture is causative rather than correlative.
If forced at gunpoint to trust one person in my supply chain, I would instantly choose the Taiwanese-American in San Jose over the mainland-Chinese in Shenzhen, every time. Likewise, I think exactly the same Indian employee will be more productive if physically transplanted to Silicon Valley and surrounded by assimilated Indian-Americans than if set up in an office in Bangalore surrounded at all times by other Indians and Indian businesses. And equally likewise, I have noted a large difference in my own work productivity doing similar tasks under different employers, or even in different work groups under the same employer.
So it isn't even national culture, racial culture, or regional culture, but also corporate culture. Breaking news on the Obvious Channel: social apes found to be heavily influenced by their peers.
Ancestor post was obliquely referring to the oft-reported tendency of mainland-Chinese manufacturers to try to cheat their customers. They have been caught copying protected or trade-secret IP, counterfeiting, substituting cheaper out-of-spec parts, and inserting hidden hardware backdoors. Outside of China, engineering/construction companies tend to import Chinese subcontractors rather than use any local labor. That's all just PRC-Chinese business culture. If you pluck individual employees out of a Chinese firm, and hire them into a multinational based out of another country, they will be perfectly fine, aside from the slightly elevated possibility that PRC intelligence will aggressively recruit them as an asset to steal exploitable business secrets. If you can manage to pull everyone they care about out of the reach of the PRC, they will likely be a great investment for you.
Clearly, the national-political-business culture of mainland PRC China is hostile to businesses that have significant interests outside of China. The government there is racist, controlling, and mercantilist-protectionist, and the businesses with a physical presence have to fit in to that culture in order to exist, along with the traditional system of guanxi social capital. It makes them difficult to work with, but as long as the price is right, it's still worth it to try. You just don't give them access to any secrets, and make sure your QA department is full of no-nonsense hardasses. And as mentioned elsewhere in this topic, that means you may have to measure total mass and moment of inertia around multiple rotational axes in order to find any unauthorized modifications that might expose you to company-bankrupting liability in your business culture. You can't "trust, but verify"; you might just have to check every unit of every shipment.
As for culture culture--wushu martial arts, the action movies including it, silk, noodles, jadeite carvings, fine porcelain, traditional architecture, religions, herbalism, the woo-woo pseudo-religion attached to the medicine traditions, etc.--Chinese culture is indisputably great, especially considering that it's what's left after the PRC government tried to annihilate most of it as part of their revolution.
There are many important and worthwhile discussions that our empirical tools are not able to significantly contribute to.
That's not to say that there is a such thing as single-dimension "greatness" metric. However, outlawing value judgments is going too far in the other direction. Of course, such things can be quite sensitive, and should be done thoughtfully.
It is called business. The company's ONLY duty is to bring profit to its investors and management is legally bound to maximise it. Now, the definition of profit may differ as well as what is profitable and what is not, but the company is legally obligated to work to bring profit and if you throw a good deal because you don't like it you better explain it to your investors.
Whether you like what your supplier does or does not is of no concern. If couple of units can be detected and written off it is treated as cost and you move on to decide if it is still profitable and whether you can get better deal somewhere else.
You let you emotions rule and it just means you are not fit for the job.
"There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company."
I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.
he's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive.
I can't believe the parent comment is so brazen. It makes me physically sick.
But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.
like shoplifting. Most large chains know exactly how much "loss" they experience, and they just raise their prices to cover it. Theft and fraud is a part of life unfortunately...
This problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.
Honestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one?
Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.
Maybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)
The product is different, but the problems are the same.
* They drop changes and problems at the last minute, so you're over a barrel with your customers.
* Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise.
* Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far.
* The new supplier will 100% play the same games.
* Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way.
* Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business.
* Are you going to admit to your customers and bosses that your products were faulty and you knew?
It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.
> * Are you going to admit to your customers and bosses that your products were faulty and you knew?
If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical.
When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice.
Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure.
All this tells me is that Made in the USA is more valuable than I once believed.
I think you’re right about the value of Made in America and societies where honesty (even about problems or risks) is valued.
The book I referenced included a number of instances where factories actively undermined individuals so that they had very little choice than to either go along with the situation, or press the ‘nuclear’ button and scupper the entire arrangement, including their own livelihood and career. That’s a tough choice for people to make.
Or: you can expect this kind of shit from pretty much any supplier. Even if you go to a western company, they all source their components from China anyway.
Also it forces their customer's customer to essentially subsidize production cost.
Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage.
As a result this immoral company externalizes aspects of their cost structure to greater society.
I've heard very similar stories from people in the card industry before. I don't think it's made up, unfortunately. Card readers from China are systematically untrustworthy.
Using the technically incorrect term "angular momentum" rather than the more correct term "moment of inertia" made me think the same thing. I would think a person tasked with building such a device would know their physics well enough to use the right term, but I may be wrong.
> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
You mean moment of inertia, not angular momentum.
You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.
For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.
In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.
Another implication of the parallel axis theorem is that the attacker could perfectly mimic every moment of inertia by shaving plastic. They wouldn't have to know which two axes were being tested because there are only three real numbers worth of information in the system to begin with (once center of mass and total mass have been dealt with.) In the whole MOI tensor there are only six free numbers which sounds like a reasonable number of parameters to fix by adding and removing small amounts of material.
What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!
> What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start
You're assuming that there were feasible alternatives; from their comment below:
"wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China. ...
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
...
Fortunately this only lasted for few months until it was dealt with"
Business lesson to include a 'no China' if the hardware handles anything sensitive, with a large fee for violating it for the middle man servicing the contract.
Another problem is anti-tampering measure is applied before the initial tampering check have been applied. Anyway, that wouldn't solve everything considering that if the chip itself is tampered with, that's undetectable short of an electron microscope analysis and even that wouldn't solve the problem of backdoor in the original chip design.
As you say, that's un-winnable. The only way to really build trust is the capacity to sue your manufacturer to oblivion - I mean the real oblivion (destruction of shareholder value) with criminal charge for the company officers, not the lame single digit percent of a single year of profit with a discount if you settle.
I reckon this could be addressed with contractual guarantees in exchange for doing business in the first place. Make it a contractual requirement that the supplier agree to adjudication in the United States in the case of security related concerns or other tampering with the final product or with the designs. They could still refuse if something happens, but I reckon at that point you can just work to shut them out entirely from the US market.
At this point, I think it's fair to establish an industry watchdog group that works as a clearinghouse for knowing all the American companies sourcing parts from the PRC and which companies they source from. This watchdog organization would be responsible for blacklisting suppliers from the entire US market if they don't agree to adjudicate in the US or if they fail to meet that contractual obligation).
This won't bankrupt them outright, but it will sever them from the US and if this watchdog group expands to aid non-US companies, could help shut them out of the entire global market.
I would also require registration of all top executives of these corporations so that you would make the punishment sticky if they moved to other companies. Overall, you've got to set a high punitive cost to non-compliance and non-cooperation.
You can't lock China out of the market without diplomatic implications. Even assuming you managed to get a purely private boycott of China together without somehow violating anti-trust law, by the time it started making a serious dent in the market the Chinese government would have started making representations to the US State Department. Where it goes from there is anyone's guess.
I honestly did not know about that. I thought that if you move any mass (remove non zero mass and place it somewhere else) there must be at least one axis which you can use to detect the change in moment of inertia.
If the mass had to all move to one other place this would be correct (the center of mass would have to change), but the attackers are able to move the mass to multiple other places.
I enjoyed your comment. But you are looking at the difference in the inertia tensor of an adulterated board from a non-adulterated one. Not the inertia tensor itself. Signal to noise problems with the measurement.
Seriously, why are we still outsourcing chip manufacturing to other countries? Sure it's cheaper, but we sacrifice a lot to have a society of corporate slaves build our tech. Security, core domain knowledge, capability, corporate secrets, patent rewards and enforcement, etc... All of it you throw away the minute you ship your manufacturing out of the country. I've seen enough board printing machines out there to start working on our own. As a country, we need to close this gap, more automation and capability and there will be no need to outsource circuit board printing and manufacturing. We will be much better off.
Canadian steel is considered by this administration to be a national security risk. But Chinese made boards and chips installed in weapons systems and crucial data centers? No problem.
The thing being called a national security risk is a lack of domestic production capability. No one is saying Canadian steel is sabotaged or something.
Yes, obviously- but one of our closest allies shorting us on steel supplies in a time of crisis has never happened, while espionage from China-produced tech is happening right now. Which should be our priority?
I love a good Trump bashing moment as much as the next guy, but this is inaccurate. The DoD has stringent requirements and quality control procedures in place for their chip procurement. Not to say they couldn't be improved, but the DoD has been aware of this threat for a while, and seems to be mitigating the risk fairly well.
Also, as far as I understand the argument, it goes beyond "Canadian steel is a national security risk". A couple of years ago, Mexico was caught laundering $2B of Chinese aluminum to avoid US taxes.
If the source is China, in a hostile period they might stop sending it. But according reportings I have read, the US DoD only uses about 0.5% of steel in the US, and only ~30% of US steel is imported.
Their argument is that you need strong domestic steel industry to build tanks, ships, etc, in case of war. Not saying I'm agreeing with it just pointing out the stated rationale behind the tarrifs.
The concern was the depression in prices was cratering local high-quality smelting capacity.
However, the DOD issued a memo (penned by Mattis iirc) indicating that there was no supply related concerns. The lack of aluminum used to justify the tariffs was in fact just the result of the LME's anti-market-tampering rules creating an incentive for metals traders to stockpile the material in private stores rather than in LME warehouses.
There was a nice article about this yesterday, but I don't have the link.
It doesn't. They claimed that so they could enact the tariffs, otherwise it would be a WTO illegal tariff. I think Canada/others are arguing that it is not a security risk and therefore is indeed an illegal tariff. This is what I remember from some articles. Please correct/elaborate.
Maybe because it undermines fundamental US business interests, and therefore wellbeing? If so it still seems like a stretch to me. Doubtful we'd have no where else to source steel if war or disaster struck.
Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?
The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.
The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.
>Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?
It's certainly easier to enforce laws and observe manufacturing processes at domestic factories than it is at factories thousands of miles away in a country that actively encourages IP theft and other wrongdoings, don't you think?
Not really, no. Why do you think it be easier to police? In concrete terms: who do you expect to do the policing? Because this is the kind of work that requires an expert, to catch an expert, messing with a product created by people 100% ignorant of _what_ they're making. Just _that_ they're making them. Where are you going find enough experts to check the work of even a single production line?
Because "thousands of miles away in a country that actively encourages IP theft and other wrongdoings". It makes perfect sense. If you don't think that, try to run any business in another country. It's just obvious that it would be easier to do domestically. You just list other set of problems. Those problems only get compounded if we'are talking about another country.
You can prosecute someone in your home country and even change policies with enough political support. Best of luck doing this in China without getting 're-educated'.
But note that as always: that's too late. By then, the damange has already been done, and the person you take out of the system is simply replaced by anyone else. Card skimmers sure as hell haven't gone away just because we wised up to them: we now just pay A LOT MORE on law enforcement, and all that gets us is law enforcement now at least staying in lockstep with the skimmer's increased sophistication.
Now imagine how much it'll cost to stay in lockstep when the level of sophistication becomes "a single component the size of half a grain of rice".
Not really: I'm letting the practical dictate the process. Catching this kind of inflitration requires experts to scrutinise the work of people who know nothing about what they're making, to catch experts with a much more vested interest in success, and you aren't going to find enough experts to take that job, at a good pay, to check the output of even a single plant. Let alone an entire industry.
If the question is "why don't we move chip production to the US, so this doesn't happen" the answer is "because you might be able to do that for one or two plants. And it'll probably drive up the price by an order of magnitutde, so they'll go out of business soon after. And you sure as hell can't move the entire industry because of prohibitive cost and shortage of manpower to do the bit that you want tacked on that can't be trusted to be done abroad".
It's hard to compete domestically against low prices caused by China's completely different standards for wages and human rights for laborers. Plus, here we don't like the idea of manufacturing industries being subsidized by the government (except in the case of "defense" of course...), while China obviously has no qualms with doing so.
Business culture is different in the US and China. If you tamper a chip in China and go unnoticed you are considered smart & the winner. If you do so here you'll be getting a more negative reaction.
Western manufacturers have a tendency to buy from the cheapest supplier, even if it isn't physically possible to manufacture at that price point without cutting massive corners or making profit off of some externality. However, when Chinese manufacturers build their own brand, you'd be surprised how the business culture suddenly creates a different set of incentives.
Trust alignment of incentives, as opposed to hey-they-promised-an-actually-impossible-pricepoint-on-paper.
I think I saw this on another thread. You're saying if someone bids that they can do something for a price that's not reasonable, it's my fault for believing their mischaracterization? "Fool me once, shame on me."
I much prefer the other mode of operation: "Fool me once, shame on you. Fool me twice, shame on me."
You're conflating a few things there. Whilst companies like to make noises occasionally about saving the planet, doing the right thing, making (your country goes here) great etc, it's just horseshit. They exist to make money. That's it.
Very interesting methods you used to detect the changes! Out of curiosity: was there a reason that taking an xray of the devices was not an option? Industrial/veterinary xray machines can often be had quite cheaply...
This feels cargo cultish. Products drop from the sky. One day they become poisonous. You have no idea how to reproduce them locally. So you come up with hacks to make then less dangerous.
We really need to get back into manufacturing if this is our brave new world.
Get an x-ray machine? They are surprisingly cheap pieces of hardware if you are willing to deal with a small area, low penetration image. Low penetration means no lead, which makes for something that's about as cumbersome as a large bar fridge.
It's mentioned in the article that X-Ray didn't help much: 'Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment'
This comment is specific to the parent talking about their experiences producing credit card terminals that ended up with PCBs implanted in them. Here it is appropriate.
And as soon as they know you are using X-Ray to test ( I would be surprised if there were no insider information leaking on the OP's end ), they would develop a way so that somehow the X-Ray images looks identical. May be a heat spreader like metal as cover up.
For the folks commenting below that we should bring the manufacturing back to the US, why wouldn't the bad guys just start bribing American workers to insert the attack hardware into devices made here?
It's not like Americans are somehow above being bribed.
The bad actors could be brought to trial in a United States court, which is a level of deterrent not included in offshore manufacturing. If a US manufacturer was found selling tampered chips then the company itself could be held liable. This would create a general pressure to increase onsight security.
There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.
Americans can be bribed to do anything. However, you don't really need to bribe or ask someone to betray their country and fellow citizens when it's done on Chinese soil...
It's possible, but it's much, much, much easier for an American company to hold another American company accountable when something like this happens. Instead of having to go through all those hoops, you sue the pants off the manufacturer.
I think people here are just suggesting that the folks who believe producing things in America would avoid a culture of abject amorality and greed seem naive, at best. Even ignoring the current obvious examples, Americans frequently engage in profitable criminal behavior: https://www.businessinsider.com/mcdonalds-monopoly-game-rigg...
This reminds me of a story about, IIRC, Soviet intelligence personnel determining that a photocopy machine at their consulate in the USA was bugged by the CIA by measuring it's weight and comparing that value to the standard value published by Xerox.
is measuring the moment of inertia not standard? I think any mechanical engineer would suggest it for that kind of problem (disclaimer; I've suggested it before for the same problem)
Why don't you guys consider to expose this by suing the bad manufacturer? I believe this could help other truely honest manufacturers both in and outside China to beat the wrong doers.
The Chinese operation would simply shut down and reopen under a different name. And the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.
But I don't think it's that good to keep playing the rat-cat game.
> the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.
Letting the bad guys keep doing what they do while putting it's own customers under shadow, to me, it sounds like the credit card companies are helping the bad guys.
As a Chinese myself and a credit card user, I'm more worried than you guys do, because we are more likely to receive that kind of treatment (compromised credit card and computer chips etc). I really hope somebody can teach those bad doers a unforgettable lesson.
Huh? This would be between you and the Chinese manufacturer. If you found s tampered unit, you don’t pay them. How should that make you go out of business?
They need the units from somewhere, with a deadline determined by their own runway, and presumably only some of the Chinese ones were compromised. Sure, they could avoid paying, but then they have to spend months getting the production run going somewhere else .. which might also be compromised.
I do not for one minute believe, that china, whose production floors are spotless, watched like a hawk and every product tested, did not know this was being added. They would have been able to tell themselves by weight. Yes, product weight is used for shipping manifests too.
This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.
What good is angular momentum when the producer can have fluctuations in its supply chain? Yes you can see when devices are not the same, but what if that happens all the time, legitimately?
Gosh, I would love to read everything about this. I know there are some videos about anti-tampering card readers on youtube, but not on the feedback race between hackers and security.
Maybe a setup to measure inertia tensor and center of mass (in that setup's axes) will be easier and I think it's what your call "measuring all angular momentums".
Was the secret service contacted? Card skimmers are a big no no. Family friend works for Dept of weights and measures and finds skimmers all the time on gas pumps. Scary stuff.
Thanks for sharing this story, and I hope you aren't put off by the huge thread of people second-guessing your competence. We need more of your kind of story.
x-ray would have been easier, as others have said.
but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.
in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).
The main lesson I learned from my Dad’s employer (twice) was always have two vendors. You can play them off of each other. When I did contract work I saw how powerful getting out or putting for away the checkbook can be. Large vendors ignore you if the checkbook isn’t moving.
I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.
Sorry, but this doesn't sound as true or there is huge mistakes done in choosing supply chain for such sensitive matter.
How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?
How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).
How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.
What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?
Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.
While it may sound sensational this was more of an operational issue, really. We were told by Visa and Mastercard that it is not even a question if we are going to be targeted. If you work in payment card industry you are constantly being attacked and the only way is to make it part of the process to deal with those things. Our network was hacked but what was important was tight, almost mathematical processes around protecting very specific material like credit card data and PINs.
For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.
It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.
Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking about malicious implants. Just google for it, it’s massive
Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.
Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab.
Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.
Presumably.
I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.
Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).
They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.
We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.