
Computer Virus Cripples Several Taiwan Semiconductor Plants - dosy
https://www.bloomberg.com/news/articles/2018-08-04/tsmc-takes-emergency-steps-as-operations-hit-by-computer-virus
======
georgeburdell
For those of you thinking that TSMC only gets set back by the time it took to
recover the equipment from the virus (1-2 days), realize that there are some
critical steps in the manufacturing process that require completion within X
time or the wafer is trash. Also realize that wafers take several weeks of fab
processing, and this may be a substantial hit to their output for a while.
They might even need to rob wafer capacity from smaller companies to placate
the tier 1 customers like Apple.

~~~
oh_sigh
What trashes the wafers? Is it some intra-wafer chemical reaction? Atmospheric
interactions?

~~~
avs733
So first, to put a term to what the grand parent is describing, they are
generally referred to as 'queue times' referencing the wafers waiting in a
queue.

Effectively it is an atmospheric interaction, most commonly with oxygen but
also light, hydrogen sulfide (from agricultural production - land is cheaper
in the countryside), and other sources of contamination.

Most wafer processing - especially depositions of materials like metals - are
done in a vacuum. if you deposit a metal (lets assume aluminum) in a vacuum
you effectively get a pure aluminum layer. Pure aluminum is highly reactive
with oxygen

As soon as that layer is exposed air it oxidizes - same thing as rust. You
know this as the dull surface finish that raw/bare aluminum gets if left out
or when you purchase it. It's what happens very quickly after you machine
aluminum - the shiny surface goes away. That is the aluminum oxidizing.

Aluminum oxide has different electrical properties than pure aluminum,
different enough to fundamentally affect the function of the transistor
devices being constructed. If I have a 2" thick piece of aluminum that layer,
maybe only a few microns (assume 5) thick after 24 hours, is basically
inconsequential. It represents 0.01% of the thickness of the piece. It is
error but not much more in the electrical properties of the piece. However, if
semiconductor layer is much thinner... maybe 10microns thick...the effect of
the oxidation on the electrical properties is much much higher and pushes it
outside the tolerances the device can have and still function.

And because the deposition layers aren't just surfaces, but penetrate to fill
trenches in the devices they are basically impossible to remove at most steps.

~~~
mirimir
So why don't they hold the queue under vacuum? Or under some inert gas?

~~~
avs733
expense of retrofitting. there is a standardized 'FOUP' that wafers are stored
in and making a vacuum capable FOUP and redesigning the systems to support it
for just a couple areas of the FAB would be hard and expensive

Edit to add a few more details:

A vacuum capable FOUP would be much MUCH heavier, requiring retorfitting of
the overhead vehicles that transport wafers aroudn the FAB. further, each FOUP
carries 25 wafers. A FAB can easily put out 30k wafers a week...it's a lot of
FOUPS. even if you just used them at critical steps it would get heavy and
expensive. I believe there was some talk abotu inert gas purges with the move
to 450mm wafers but, like 450mm wafers, that never happened.

~~~
mirimir
OK. But maybe plastic bags and tanks of helium?

~~~
froindt
I spent some time working in a fab. Nothing is cheap, competition is tough,
yield is everything. 99.8% yield on each step of a 500 step process is good
for 38% overall recovery.

These events have been shown to be very rare, and the costs would greatly
outweigh the benefits (for the time being).

The cheapest piece of equipment was the stand the FOUPS come down and rest on.
From memory, that was easily 5 figures, perhaps low 6. Every other piece of
processing equipment was insanely more expensive.

Queuing theory is an active field of research with great applications in how
wafer processing. The total process has to be looked at. Wafer priority, wafer
value, planning preventative maintenance on the machines, etc. could all be
taken into consideration when scheduling when wafers will run.

Retrofitting every "robot" that grabs FOUPS for a much heavier load, upgrading
the tracks if needed, replacing every single FOUP, adjusting factory flow and
inventory holding areas, doing qualifying trials on the FOUPS (to make sure
they're functioning correctly and not leaking), etc. are considerable expense.

Prioritization of wafers and timing the preceding step so the required machine
is certain to be ready in the window would yield better results from a strict
business ROI perspective.

~~~
mirimir
> These events have been shown to be very rare, and the costs would greatly
> outweigh the benefits (for the time being).

Yes. And I was responding to georgeburdell's comment about long-term losses
from this virus infection. In that maybe some crude hack could ameliorate
them.

> For those of you thinking that TSMC only gets set back by the time it took
> to recover the equipment from the virus (1-2 days), realize that there are
> some critical steps in the manufacturing process that require completion
> within X time or the wafer is trash. Also realize that wafers take several
> weeks of fab processing, and this may be a substantial hit to their output
> for a while.

~~~
froindt
Sorry if that came off abrasively. avs733 mentioned everything is on a
different scale in semiconductors and I forgot how long it took me to grasp
that.

In school (industrial engineering) we talked about length and thickness
tolerances being quite tight at 1/10,000 inch. The units used in
semiconductors are nanometers, angstroms, etc. You might do a process and lay
down just a couple atom thick layer of material...I had no idea that was
physically possible.

~~~
avs733
I do enjoy when we are all nice to each other.

Semiconductors still blow me mind...eventually we will run out of base silicon
to use and are possibly really fucked but its been a fun ride so far.

~~~
froindt
The thing that shocked me the most is that it's economically viable. The
tolerances are so tight, the machines so expensive, the volumes so big, that
the math works. If I had no idea what semiconductors were and you just sat me
down and started explaining things, it'd seem like magic that it would even be
physically possible, much less economically...but somehow it is.

~~~
avs733
It is the most bullshit sounding industry ever to an outsider. You make things
smaller than the wavelength of light, out of sand, to turn on and off
trillions of times a second, to do math, to make all the world's information
instantly available on a metal and plastic block I keep in my pocket?

No, thats not real, that's a silly science fiction novel.

------
11thEarlOfMar
It's surprising to many, but these tools run primarily on Windows, and,
without anti-virus protection because it can interfere with critical timing
actions that the tools need to make. The IT systems are locked down like a
fortress, but tool technicians can still bring viruses in and transmit when
they connect their laptops for diagnostics and maintenance. I can easily see a
virus running amok.

~~~
rectang
We've seen this before many times. It's an inherent weakness of proprietary
software solutions: you're at the mercy of the vendor as far as porting it to
a new platform. The weakness becomes more urgent as the platform grows
old/unsupported/insecure/etc. It may have been Windows this time, but it could
have been DEC or Solaris or what have you.

Open Source Software may or may not be easy or economically realistic to port,
but at least the users have a say.

~~~
robbyt
This isn't caused by lack of open source software, this is caused by lack of
management, partitioning, and containment. I assume the issue was caused by a
100% flat network, with office worker's laptops being on the same subnet as
million dollar fab machine computers.

~~~
jdietrich
Even an airgapped system can be breached, as we saw with Stuxnet. There's no
perfect solution for protecting vulnerable systems.

~~~
delfinom
Remove all data ports?

Upload instructions via punch cards

~~~
MertsA
"Hey guys I found these punch cards in the parking lot"

------
0x0
So it is possible to get malicious code execution on the fabs that build the
next generation of processors for our devices. What are the chances that a
sophisticaled stuxnet-like attack inserting backdoors in the CPU design will
follow?

~~~
kingosticks
It would be much easier to do this at an earlier stage - before the design is
handed off to the foundry. Modifying masks for even trivial changes is very
difficult and that's assuming knowledge of the circuit (which your malicious
code probably won't have). Plus I imagine the verification tools would spot
the difference, flagging it as an imperfection which would then attract
scrutiny. I think this is extremely unlikely.

~~~
MertsA
Some backdoors can be almost impossible to detect with any kind of traditional
wafer inspection tools. E.g., it might be possible to transparently cripple
something like Intel's RdRand instruction by changing only the doping on some
of the gates inside the hardware RNG circuit. It would look completely
identical under the best microscope and looking at the output wouldn't give
you a good indication of whether or not you had a broken RNG or a good one.
The design of Intel's RNG involved passing the output of the RNG through some
whitening and IIRC AES so it might look completely random and pass every
statistical test but the hardware entropy source might only realistically have
10 bits per second of real entropy.

I still think it's unlikely that someone would go through the effort to insert
a super secret backdoor by surreptitiously modifying the masks but there are
some interesting techniques that can make finding a backdoor almost impossible
if you don't already know about it.

~~~
kingosticks
Sounds like the really cool "Stealthy Dopant-Level Hardware Trojans" paper
from a few years ago. I'd forgotten about that, well worth a read for those
interested.

However, you must remember that as well as precise knowledge of the circuit,
anyone other than an insider has close to zero chance of getting a doping
change correct. Even an insider's chances are very low.

And then your high-level testing is in place too. It's not common to have
circuits where BIST can't give you full confidence but it happens; for things
like the RNG here, also due to test tool limitations etc. So you must be doing
both both general and highly targeted post-manufacturing testing also. The
extent of that testing may depend on your market.

------
ohiovr
What is the motivation of the crime?

~~~
discordance
Possible bad actors and motivations:

\- Competing fabs \- Competitors to Apple \- China (nationalist sentiment or
something to do with the recent IP theft court cases)

------
jplayer01
Well, rip prices for whatever they're manufacturing.

------
j-pb
Am I the only one bothered that the largest pure play fab in the world is
degraded to an apple supplier. This has consequences far beyond apple.

~~~
xevb3k
Yes, the connection is weird.

Prompted me to look up the numbers though. Apple could buy TSMC for cash, four
times over... pretty incredible.

~~~
baybal2
No, they can't. Nobody will sell them TSMC, and even if they will, Chinese
govt will shut that down.

~~~
EliRivers
Just to check, are you saying the government of the Peoples Republic of China
will shut that down, or the government of the Republic of China?

~~~
chrischen
Taiwan is the latter. Taiwanese people consider themselves Chinese (both
ethnically and nationally).

~~~
tptptptptptp
I call BS on that

Regarding the 2019 Taiwan Independence referendum, the Foundation’s data shows
that 20.7% of the population are incredibly in favor of the vote and 28.8%
generally approve, indicating that over half of the population support the
referendum.

[https://www.taiwannews.com.tw/en/news/3385076](https://www.taiwannews.com.tw/en/news/3385076)

Most Taiwanese consider Taiwan, China separate countries, poll suggests

[https://www.scmp.com/news/china/policies-
politics/article/20...](https://www.scmp.com/news/china/policies-
politics/article/2099286/most-taiwanese-consider-taiwan-china-separate-
countries)

Nearly 70 percent of Taiwanese are willing to go to war if China were to
attempt to annex Taiwan by force

[http://www.taipeitimes.com/News/front/archives/2018/04/20/20...](http://www.taipeitimes.com/News/front/archives/2018/04/20/2003691661)

~~~
chrischen
How do you call BS? Yes of course they consider the two separate countries.
The issue isn’t that. The issue is which country is the historical
continuation of “China” after WWII, and that’s not even what I’m talking
about. I’m talking about ethnic and national identity; the argument over which
government of China is legitimate was not part of the conversation.

Taiwanese people are Chinese people, as is much of Singapore and Hong Kong.

I’m not sure what your referendum links have to do with anything.

------
theshadowknows
Bets on whether Intel is behind this?

------
Keyframe
In other news: "Computer Virus Cripples Nvidia Chipmaker TSMC Plants"

------
JudasGoat
Now if the headline had said "AMD's chipmaker crippled by virus", I would have
suspected Intel of industrial sabotage. Of course less people would read that
title I'm guessing.

~~~
nv-vn
There is honestly no way Intel would be stupid enough to do that. I hope.

~~~
rasz
Definitely not Intel, but remember those CTS Labs clowns engineering media
campaign touting fake cpu flaws? [https://wccftech.com/report-alleges-amd-
ryzen-epyc-cpus-suff...](https://wccftech.com/report-alleges-amd-ryzen-epyc-
cpus-suffer-13-fatal-security-flaws/)

Wonder who shorted AMD/Nvidia recently

