That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.
This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.
The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.
> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.
LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/
> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.
At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.
After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.
Management networks need rack level isolation.
Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.
cool, thanks for that info.
> just to write off the hardware
maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.
at the expense of a dedicated switch.
But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To me, that makes it sound like they could download from a remote host and inject code and do literally anything.
5. When a server was installed and
switched on, the microchip altered
the operating system’s core so it
could accept modifications. The chip
could also contact computers controlled
by the attackers in search of further
instructions and code.
The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.
Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.
The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?
For scale, this alone is about the size of a large SMD capacitor and would basically be lost in most designs today.
For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.
They probably found it out when they were repeatedly tried to reflash the BMC flash, and saw that checksums did not match.
And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"
Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.
Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.
The recovery overrides the primary if detected by default.
The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.
That makes the whole thing gloriously simple.
A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.
It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.
So, they would've been screwed even if they were doing board testing outside of China.
That's a clever trick.
But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.
Left of the sata connector. An empty space with 8 pads for an smt eeprom or flash. It is occupied by the thingy on bugged boards.
Right below is the Aspeed chip - the BMC
"Basically, it's a perfect spying platform. You can't control it. You can't patch it. It can completely control your computer's hardware and software. And its purpose is remote monitoring. At the very least, we need to be able to look into these devices and see what's running on them."
Sure you can. OEMs regularly release patches for platform BMCs.
Do you know this, or are you speculating?
"The illicit chips could do all this because they were connected to the baseboard management controller..."
"17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."
That’s an interesting choice.
Where each unit was planted with a image recorder. And for years, the American spy agencies had a great laugh, that they were able to intercept all the documents that the Russians made a copy of.
Back then, this was an off-network infiltration. Where the copied images, were retrieved during regular servicing intervals by a Xerox technician.
edit: whoops, looks like amatecha beat me to posting more info
In the developed world, hatred is often manufactured to gain power... The leaders of a nation can cause more harm in their quest for glory than the average citizen of an opposing state.
Also, security through obscurity is, as we know, an illusion. Information always finds a way out.
I understand your point, but there should be limits.
Judging by the number of parts ordered from Xerox, Zoppoth believes that spy cameras may have been installed in photocopiers all over the world, to keep an eye on U.S. allies as well as enemies
Our team promoted liberty and democracy worldwide (see below). The other promoted totalitarian dictatorship, labor camps, etc. If your team is liberty and democracy - self-determination for all - then it's not 'our team', it's everyone's team.
To address the elephant: Yes, there were many, many exceptions to the West's support of democracy, and many of them were awful (Indonesia, the Mideast, Zaire/Congo, and many others come to mind) but beyond a doubt, during the Cold War and after democracy and liberty exploded - stop and compare it to any other time in human history. Lots of bad things happened, but compare that with Soviet-dominated areas such as the USSR itself and Eastern Europe.
Well played, Bloomberg. Well played.
Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).
They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.
We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."
I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.
The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).
If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.
And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it.
One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets.
The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.)
If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?
All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.
Say their profit margin is 1%. If you offer to pay them a 1% higher price, you're offering to nearly double their profits on whatever they sell to you.
Admittedly, that only works if you are a sufficiently large portion of their business. If you're a tiny percentage, then it may take a lot more to motivate them if their other customers aren't willing to pay more.
Madness if you ask me but there we are.
IMO this is not indignation, it's supplier sourcing 101.
> If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
I will definitely agree that holding suppliers to standards regarding consistent output, unadulterated products, and conducting audits all make production much more expensive.
But if you're a device manufacturer, these sourcing controls are key to shipping quality products.
Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).
So, realising they could copy our hardware, but didn't have our software, they responded by trying to hack my servers, multiple times, from the same IP they sent manufacturing data from. A quiet word with their management would stop it, and it'd start again a couple of days later.
These people have no shame, and if we are going to go for lowest cost at all times it is what manufacturers should expect to happen.
How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.
Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.
It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way.
Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?
after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.
I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened.
However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.
I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.
They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.
> they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them.
I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs.
And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify.
You could also be mandating how many boards are allowed or that the SNs go on early in the build process.
When you can't even trust that food is real, and when you have to bribe even low level provincial government employees (can you imagine needing to bribe DMW workers or even police?) - what else is an average person to do except to treat it as the norm in order to survive? Unless you're an elite, the only other option is to leave, and not everyone has that choice. Very little is considered wrong over there, aside from criticizing the powers that be; as long as it helps your clan. It doesn't help when Western companies and governments look the other way, as we see in past news stories and even the comments here.
I'm not condoning the behavior; just explaining.
From a naive Finnish perspective you could see American business culture as dishonest. People lie all the time and promise things that are false and you need massive amount of legalese or they fuck you over. When they say something, you can't trust it as much as you can trust a Finn (or Swede). American way is to lie to your face in pre-negotiation but then be truthful to the legally binding agreement. Germans or Belgians are more truthful beforehand and they stick to deals better without legal enforcement looming over them, but require more detailed written deals than Finns. Who is right? Should we punish Americans for their behavior or adopt their culture?
Of course not. You must understand that that the level of acceptable exaggeration, overpromise and bullshit and trickery is strictly culture dependent and relative. Someone is dishonest if they "lie this amount above average" in their cultural context. After you make an adjustment, you know how to get the truth.
Chinese manufacturers have the capability to stick to standards, take responsibility and deliver high quality stuff. It's just more dependent on personal relationship. Just placing an order has less trust as default. Either you develop "quanxi" with your suppliers or work trough some third party that understands both worlds. Third party understands what you need and knows how to get it from the Chinese suppliers because they have a relationship.
Chinese pay the cost that comes with their business culture. They can't network as fast as you can in open western cultures with more default trust between strangers.
Contrary to the impression your post gives, the EU has much more "sanctions" against food (esp. imports) than the US does, and is generally taxing much heavier and restricting many more things. Importing essentially any milk product into the EU is only possible on an exception basis. Trump has a LOT more work before extra sanctions in the US will become comparable to the EU ones.
What is often not said is that the EU not only has sanctions on its borders against food products, it actually has internal sanctions and regulations that are purely political in nature.
For example "champagne" in the EU can only be made within a particular region in France. Which leads to funny, in a sad way, consequences:
This applies to many products. Wines, cheeses, cakes, fruit, meat, potatoes, candy ... and is generally absolutely ridiculous:
For your other part of your comment, I am unwilling to give a real answer. Obviously the US checks it's food quality. It's made different choices than the EU, famously the US allows hormone "enhanced" meat, but it does check food quality and safety. There are heavy penalties for breaking them.
Did you look at the map? It's nearly all things with a place name in the product name. For example, "Welsh Beef". Why is it reasonable for a consumer to buy Welsh Beef and for it to possibly have come from somewhere other than Wales?
It seems to me, more like generalising the concept of trademarks to place names rather than just company names and brand name.
Not all cultures are great. If you are claiming “all cultures are great”, then perhaps “great” has lost any useful meaning.
Discrimination, as the ability to discriminate features to identify useful pattens, is not *-ist; it’s a critical skill that is being sullied by some neo-intellectual BS that passes for “equality” or something.
The ability to identify patterns of behaviour prevalent to a class of suppliers and avoid them, rewarding other supplies — even from the same area of the world — who are better; even “great” perhaps, is a valuable skill...
But perhaps I’m wrong. They’re all great. Carry on.
It's the norm in the west to not conflate culture with race. But to think that this is the norm in the rest of the world is to project your value system on others.
If we define "a culture" as set of sociological customs, value systems, expected behaviours and stimuli/response criteria;
and if we define "culture greatness" as such sociological customs, value systems, and behaviours that lead to specified and asserted goals;
than one can reasonably compare and contrast cultures in meeting (or even striving) to these goals, and thus discuss their greatness.
If forced at gunpoint to trust one person in my supply chain, I would instantly choose the Taiwanese-American in San Jose over the mainland-Chinese in Shenzhen, every time. Likewise, I think exactly the same Indian employee will be more productive if physically transplanted to Silicon Valley and surrounded by assimilated Indian-Americans than if set up in an office in Bangalore surrounded at all times by other Indians and Indian businesses. And equally likewise, I have noted a large difference in my own work productivity doing similar tasks under different employers, or even in different work groups under the same employer.
So it isn't even national culture, racial culture, or regional culture, but also corporate culture. Breaking news on the Obvious Channel: social apes found to be heavily influenced by their peers.
Ancestor post was obliquely referring to the oft-reported tendency of mainland-Chinese manufacturers to try to cheat their customers. They have been caught copying protected or trade-secret IP, counterfeiting, substituting cheaper out-of-spec parts, and inserting hidden hardware backdoors. Outside of China, engineering/construction companies tend to import Chinese subcontractors rather than use any local labor. That's all just PRC-Chinese business culture. If you pluck individual employees out of a Chinese firm, and hire them into a multinational based out of another country, they will be perfectly fine, aside from the slightly elevated possibility that PRC intelligence will aggressively recruit them as an asset to steal exploitable business secrets. If you can manage to pull everyone they care about out of the reach of the PRC, they will likely be a great investment for you.
Clearly, the national-political-business culture of mainland PRC China is hostile to businesses that have significant interests outside of China. The government there is racist, controlling, and mercantilist-protectionist, and the businesses with a physical presence have to fit in to that culture in order to exist, along with the traditional system of guanxi social capital. It makes them difficult to work with, but as long as the price is right, it's still worth it to try. You just don't give them access to any secrets, and make sure your QA department is full of no-nonsense hardasses. And as mentioned elsewhere in this topic, that means you may have to measure total mass and moment of inertia around multiple rotational axes in order to find any unauthorized modifications that might expose you to company-bankrupting liability in your business culture. You can't "trust, but verify"; you might just have to check every unit of every shipment.
As for culture culture--wushu martial arts, the action movies including it, silk, noodles, jadeite carvings, fine porcelain, traditional architecture, religions, herbalism, the woo-woo pseudo-religion attached to the medicine traditions, etc.--Chinese culture is indisputably great, especially considering that it's what's left after the PRC government tried to annihilate most of it as part of their revolution.
What measures a culture's greatness? How can this value be empirically deduced?
That's not to say that there is a such thing as single-dimension "greatness" metric. However, outlawing value judgments is going too far in the other direction. Of course, such things can be quite sensitive, and should be done thoughtfully.
the value of civic participation and honesty
Whether you like what your supplier does or does not is of no concern. If couple of units can be detected and written off it is treated as cost and you move on to decide if it is still profitable and whether you can get better deal somewhere else.
You let you emotions rule and it just means you are not fit for the job.
Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.
I can't believe the parent comment is so brazen. It makes me physically sick.
But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.
Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.
The product is different, but the problems are the same.
* They drop changes and problems at the last minute, so you're over a barrel with your customers.
* Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise.
* Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far.
* The new supplier will 100% play the same games.
* Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way.
* Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business.
* Are you going to admit to your customers and bosses that your products were faulty and you knew?
It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.
If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical.
When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice.
Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure.
All this tells me is that Made in the USA is more valuable than I once believed.
The book I referenced included a number of instances where factories actively undermined individuals so that they had very little choice than to either go along with the situation, or press the ‘nuclear’ button and scupper the entire arrangement, including their own livelihood and career. That’s a tough choice for people to make.
Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage.
As a result this immoral company externalizes aspects of their cost structure to greater society.
So now he needs to make a new test. Ow that that is out of the bag
You mean moment of inertia, not angular momentum.
You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.
For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.
In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.
What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!
You're assuming that there were feasible alternatives; from their comment below:
"wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China. ...
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with"
As you say, that's un-winnable. The only way to really build trust is the capacity to sue your manufacturer to oblivion - I mean the real oblivion (destruction of shareholder value) with criminal charge for the company officers, not the lame single digit percent of a single year of profit with a discount if you settle.
This very rarely happens even if people have got killed. Also, good luck suing a Chinese company in China.
At this point, I think it's fair to establish an industry watchdog group that works as a clearinghouse for knowing all the American companies sourcing parts from the PRC and which companies they source from. This watchdog organization would be responsible for blacklisting suppliers from the entire US market if they don't agree to adjudicate in the US or if they fail to meet that contractual obligation).
This won't bankrupt them outright, but it will sever them from the US and if this watchdog group expands to aid non-US companies, could help shut them out of the entire global market.
I would also require registration of all top executives of these corporations so that you would make the punishment sticky if they moved to other companies. Overall, you've got to set a high punitive cost to non-compliance and non-cooperation.
(Ironically the other direction prevails in regard to Israel: there are anti-boycott laws! https://www.lrb.co.uk/v40/n14/amjad-iraqi/short-cuts )
Let that sink in for a moment.
They faked the data on what it could handle; and with things such as aircraft, lives depend on that data.
The theory, from the Trump crowd, is that Canada is also engaged in similar shady dealings with China. If true, that would put the US at risk.
However, the DOD issued a memo (penned by Mattis iirc) indicating that there was no supply related concerns. The lack of aluminum used to justify the tariffs was in fact just the result of the LME's anti-market-tampering rules creating an incentive for metals traders to stockpile the material in private stores rather than in LME warehouses.
There was a nice article about this yesterday, but I don't have the link.
The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.
The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.
It's certainly easier to enforce laws and observe manufacturing processes at domestic factories than it is at factories thousands of miles away in a country that actively encourages IP theft and other wrongdoings, don't you think?
Now imagine how much it'll cost to stay in lockstep when the level of sophistication becomes "a single component the size of half a grain of rice".
If the question is "why don't we move chip production to the US, so this doesn't happen" the answer is "because you might be able to do that for one or two plants. And it'll probably drive up the price by an order of magnitutde, so they'll go out of business soon after. And you sure as hell can't move the entire industry because of prohibitive cost and shortage of manpower to do the bit that you want tacked on that can't be trusted to be done abroad".
Well, this thread is exactly the reason we subsidize the defense industry to make sure things are produced in the US.
I actually agree it might be better for the Americans to manufacture things in America - especially things used in critical government systems.
But this seems like a human problem - if all the factories were moved to America, couldn't those factory managers etc also be bribed?
American manufacturing in the 60s was rife with unions w/ ties to organized crime.
Trust alignment of incentives, as opposed to hey-they-promised-an-actually-impossible-pricepoint-on-paper.
I much prefer the other mode of operation: "Fool me once, shame on you. Fool me twice, shame on me."
That's enough, full stop, say no more. The other costs are real yet they're either not marginal, are borne by others, or both.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.
I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?
They will never do that, because they look for the cheapest solution.
The bigger the company, the less it cares about things other than cost. This is why Mediatek and Broadcom can usurp the market of network SoCs, while making products with atrociously bad support. I personally dealt with both, and say that they wholely match their popular culture image.
I don't know how it is with USA, but for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres, for as long as they get them made inside the country.
I don’t know if any companies do PCB manufacturing and assembly outside of China in large numbers.
Well, if your chips are bigger and slower, you will need more chips and mounts/packaging to place them. If you need more chips then the weight of missile/plane/tank will be increased and available space decreased.
So at the end, the 'uncaring' military will receive a weapon which is worse than competition.
All thanks to an outdated chip.
But I'd guess momentum is hard to change.
It's the unknown unknowns that get you.
at first I had the same thought. but i have to question how securely the same manufacturing could be done in a US plant.
the US employee base has its fair share of desperate, ethically challenged individuals. and plenty of incentives to make a quick buck could be offered here too. idk.
OTOH, given US law enforcement's low efficiency what are the chances of being caught? and what if it's merely corporate espionage?
finally, the US is an open society with strong personal freedom guarantees builtin. what if the perpetrator has ties to a foreign country and simply leaves the US after they've installed the vulnerabilities?
Due to requirements we opted to have the only large meeting room to have outside our secure zone. This created an issue as we had no network access from there and in the end we decided to use slow GPRS terminal for the test.
The end-to-end test starts with offline transactions which by their very nature are quite fast (it is negotiated between terminal and card).
But then we went to online transaction and it finished instantly too.
The auditor, bewildered, proclaimed the test failed as he assumed it was incorrectly processed offline instead of going online. But then I pointed out to the printout to show ARQC (basically says it was certified online).
Now, the real discussion started. The terminal was very slow taking quite few seconds to establish GPRS and then even more for the SSL handshake so the auditor said it was not possible to make it work.
How it worked was that I have completely gutted OpenSSL and had entire cryptographic state stored locally (safely, using internal HSM) so the SSL session could be optimistically re-established without another handshake even after TCP connection was closed. The first message the terminal sends is already encrypted transaction message, there is no SSL handshake. I wrote an application to terminate the connection in our data center so that it stored the states of each connection in the database. The entire handshake was only done if the first message could not be decrypted successfully.
The operating system was single-threaded with no multitasking of any kind. This meant that all applications on this device did their operations sequentially. Send network message, print something, display something, etc.
I wrote a cooperative multitasking functionality into the application (using coroutines) so that it could work on multiple tasks at the same time (like talking to network and printing).
I then have segregated all data on the printouts so that it can start printing without having to already have response from network. Hopefully if everything went right, the response would come before it even came to that place on the printout effectively looking as if it was done in zero time.
But am I wrong to have my hackles raised by a) the roll-your-own security nature of this, b) the reliance on a single developer's single stack implementation as what guarantees the integrity of the system? It seems like there are a lot of assumptions baked in.
I, too, would love to see a more detailed write-up--if there's a big idea here (almost a unikernel thought), it deserves to be shared and tried by fire.
quote captures the human element playing strong in face of bad system
If the device is sealed with an anti-tampering system then the contents must be checked by a trusted entity before being sealed.
Trying to guess the contents of a box that you cannot open sounds a bit like madness.
As I hinted in another comment I suspect that they had a suspicion and checked those motherboards very, very carefully.
Use X-ray? or whatever can penetrate the exterior shell
The more critical the field, the more you have to treat those devices as untrusted before attaching them to your trusted zone.
Given how sophisticated these attacks can be, I'd think they'd issue disposable equipment to be destroyed on return, like a cheap netbook or something. I don't see how you could trust an individual viewing a simple X-ray scan to detect some extra microchip the size of a signal conditioning coupler.
Then again many companies or public institutions would find it hard to justify shredding each week maybe tens of laptops and phones that still have to be good enough to work on. Basically they still have to be a "standard issue" device with your company's software stack, config, etc.
I'm sure someone can find a good compromise between security and wastefulness.
But it's useful to know when/if you're being targeted.
Basically, any device brought to the US and a number of other countries are issued for one-time use; if they leave our custody for even an instant, they are to be scrapped.
From what I understand, Boeing does this when employees visit France.
They have had problems with men in nice suits going through laptops stored hotel safes.
> We would be getting products from China with added boards to beam credit card information.
>> Trying to guess the contents of a box that you cannot open sounds a bit like madness.
>>> Use X-ray? or whatever can penetrate the exterior shell
2 different types of attacks, 2 different types of responses.
But I think the GP's question is: "Whether it would be cheaper" - in the sense whether such an expensive QA process could have been averted by having a more trustworthy partner. One whom you're not on a race hack after hack.
You cannot just trust the word of a contractor on this because it's your ass on the line.
You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages  and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).
Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.
To give a bit of background, when you type your PIN on credit card terminal it is not the terminal application that is really getting the pin (well, except for special credit cards but that is really problem of the Bank that issued the card). The Visa/Mastercard mandate that the application don't have control over the PIN and that the PIN entry uses physically separate keyboard and display.
To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry and the PIN is transferred directly to the HSM where it is being encrypted before it is transferred to the application processor for the rest of processing.
Perhaps this sounds too dull to ask, but what stops the terminal from just ... not separating the keyboard and display?
But still as a user I have no idea if I'm talking to a certified machine or not.
I'm not so sure. "And then initial supplier inserted hardware that thwarted all those pressing considerations" sounds a punchline to me. :(.
The safeties are mainly to guard against the rest of the world. For example it prevents tampering in transit, or even in our own company - disgruntled employee can't do anything.
Or think for a second about the fact that we leave the device for, hopefully, entirety if its life at the client site. We had clients that were shady businesses like strip clubs that no other companies would touch with a stick.
This is the easy part.
The hard part I remember was establishing secure communication between all components in the system (initializing HSMs, injecting keys). I remember helping designing the process and writing hundreds of documents describing various security-related procedures like how the HSM racks are inspected, how the keys to the racks are fetched from the safes, how there are multiple safes for multiple security officers, how the officers are prevented from ever having access to other safes, how fetching anything from safes requires logging and using tamper-evident containers, how the logs are inspected, and so on.
I have designed a special cryptographic protocol so that we could generate and inject keys to the devices in KIF (Key Injection Facility) and separately to our database (to establish communication with the terminal). Fun.
We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)
(If anyone reading is looking for a job in security engineering, we're hiring! https://www.clover.com/careers/engineering)
We also had special screens created for all cameras in the datacenter to block view on the HSM racks.
The biggest issue was, just before end-to-end test we figured out we forgot one of critical procedures (it was establishing authenticity of the HSM used) and we had to scramble to get new HSM and to re-establish all cryptographic material (so new storage keys, etc.)
The general problem with most "industry security" approaches is that they simplistically attempt to wrestle ultimate Godmode-control for themselves, rather than working towards eliminating it.
The requirement for the tamper proofings is literally the first requirement in the PED standard:
A1 Vendors must comply with all components of A1.
The PED uses tamper-detection and response mechanisms that
cause the PED to become immediately inoperable and result in the
automatic and immediate erasure of any secret information that may
be stored in the PED, such that it becomes infeasible to recover the
secret information. These mechanisms protect against physical
penetration of the device by means of (but not limited to) drills, lasers,
chemical solvents, opening covers, splitting the casing (seams), and
using ventilation openings, and there is not any demonstrable way to
disable or defeat the mechanism and insert a PIN-disclosing bug or
gain access to secret information without requiring an attack potential
of at least 25 per PED, exclusive of the IC card reader, for
identification and initial exploitation as defined in Appendix B of the
PCI POS PED DTRS
I don't know this device internal and the PED/PTS exact requirement but it seems plausible for me.
You have something like a physical compartment who include the NFC and everything needed to process it like in a classical terminal. This compartment is highly secured as requested by the specification with just a very simple interface for the android part to send the amount to bill.
I've seen a lot of each-machine running on windows. Doesn't they work like this with the windows machine just managing the display buttons to select the amount and sending this information to the secure part who handle card interaction, pincode and delivery of the money ?
Now none of these certifications or standards is bullet proof but people have a very skewed vision of the PCI certification process likely due to bias of only having interacting with the PCI-DSS requirements for merchants and low levels to boot meaning they didn't had to do anything but to fill the SAQ themselves and be on their way.
HSMs are required so that the company does not need to have PIN codes exposed anywhere. Not having PINs or full credit card data makes your life easier as there is nothing to steal from you in the first place.
If your company stored PIN codes it means you were in breach of the contract and it had to lie to the auditors to pass the certification.
The pin we are talking about is what is customized on your credit card (directly in its memory) or its equivalent in your bank's HSM for the sole purpose of performing CVM step negotiated by yor card and payment terminal.
Legit companies don't want the info and anyone that wants the info isn't doing anything legal with it.
Devices that accept cards need to comply with PED/PTS security requirements including very strict physical security requirements which are validated by PCI council approved laboratories and firms.
You are not getting a device on the market or usable with any merchanet network without complying with this: https://www.pcisecuritystandards.org/documents/pos_ped_secur...
and a few other standards.
The question is, how Chinese banks coax Visa into allowing them using them.
I think the GP is confused on how a POS works, POS isn't a POI most of them don't touch the credit card they just talk to the reader, most readers today are P2PE closed loop solutions so the only thing the POS does is sends to the reader charge the next card $X the reader will then reply if the transaction went through or not and that's it.
The reader itself will talk to the acquiring bank or the payment provider in a point to point encrypted closed loop and the merchant would never see any credit card details.
Sure you can send fake devices to be certified and sell something completely different but the same can be said for any certification and if you get caught boy or boy...
It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.
Visa/Mastercard required that there are two people present and that there are two people required to open the rack.
We had it modified so that it has second lock.
The additional lock did not fit perfectly. During the procedure when I tried to close the door the door snagged on the lock and then slammed shut. It wasn't a lot of force but it caused the HSM to loose its keys and required a lengthy procedure to get three security officers to initialize the HSM with components stored on their smart cards during which the whole system was unavailable.
To echo, it's actually quite trivial to bypass anti-tamper stickers with acetone and a needle.
We really need to get back into manufacturing if this is our brave new world.
You only need to measure three angular momentums, all other can be calculated. See https://en.wikipedia.org/wiki/Moment_of_inertia#Motion_in_sp...
"This shows that the inertia matrix can be used to calculate the moment of inertia of a body around any specified rotation axis in the body."
On the attacker side, they only need to make sure three angular momentums match.
It's not like Americans are somehow above being bribed.
There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.
Modern times, but same old methods of "debugging"
The angular momentum stuff is innovative though.
> the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.
Letting the bad guys keep doing what they do while putting it's own customers under shadow, to me, it sounds like the credit card companies are helping the bad guys.
As a Chinese myself and a credit card user, I'm more worried than you guys do, because we are more likely to receive that kind of treatment (compromised credit card and computer chips etc). I really hope somebody can teach those bad doers a unforgettable lesson.
Angular Momentum of a body at rest is zero.
This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.
Can you elaborate on this concept a bit? I'm not familiar with this term and the sources I looked up were pretty physics-y and out of my depth.
This sounds like a very interesting and creative solution. Good lateral thinking on your part! :)
but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.
in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).
I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.
How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?
How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).
How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.
What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?
Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.
For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.
It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.
Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.
Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab.
Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.
I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.