
TSMC Nanke 14 Factory Production Interruption Could Affect NVIDIA and Others - IMTDb
https://www.hardocp.com/news/2019/01/28/tsmc_nanke_14_factory_production_interruption_could_affect_nvidia_others
======
sevensor
Let me tell you a story about sulfuric acid. When I was working process
integration at a fab, I had a long series of meetings about qualifying a new
source of sulfuric acid. Not a new supplier even, nothing that risky. Just a
new source, meaning the iso (tanker truck) full of sulfuric comes from a
different plant owned by the same chemical company.

What it comes down to is that it's basically impossible from a process
integration perspective. Sure, you can take assays, and you do. But what about
the things you aren't looking for in your assays? All you can do is plug it
in, cross your fingers, and look at the yield trends. Not only does sulfuric
mix in the supply lines, it gets used throughout the entire process. So if you
do have some kind of strange contaminant that you didn't test for, maybe it
only actually hurts you pre-gate. And the damage ramps up slowly and unevenly.
But if your yield tanks, then you have to scramble to figure out what
happened, and sulfuric isn't even your first suspect. (Because there are
thousands of things like this that can tank your yields.)

What's worse, you're literally over a barrel. Sometimes Plant A has to shut
down and can't send you another iso. Do you take an iso from (unqualified)
Plant B? Do you crack open a barrel of sulfuric? You have a stock of qualified
sulfuric in barrels, for situations like this, but in practice barrels are
worse than isos. Metal everywhere.

I'm not saying it's sulfuric in this case. Lots of chemicals have a story like
this one, but it would also not surprise me one little bit.

~~~
parimm
Wouldn't using multiple Spectrometry/Spectroscopy methods help in finding out
the impurities in the process chemicals?

ICP-MS and NMR should cover most elemental impurities.

~~~
Const-me
I don't have any experience working in that industry.

But based on what I know, modern ICs require exceptional levels of purity of
the materials. A single atom can ruin a transistor, and therefore the complete
chip.

AFAIK the methods you mentioned aren't sensitive enough to detect individual
atoms in barrels of stuff. E.g. ICP-MS detects 1E-15 concentrations, but in
absolute numbers, 1E15 molecules is only 1.66e-9 mole, e.g. for iron
(55.8g/mole), it translates to 1E+10 defects per kg of stuff. Way too many.

~~~
dchichkov
Wouldn't you be able to run the test multiple times and increase your
sensitivity to an arbitrary level? Too expensive?

~~~
lostmyoldone
Studied a fair bit of chemistry, but not a pro, so this is only a rough
outline.

A lot of chemical test equipment use semiconductor sensors of some kind -
optical or otherwise - and most likely the sensitivity limit is simply the
noise floor of the sensors. In good instruments they tend use good, or amazing
detectors but they are still operating at the noise floor, or sometimes even
below.

So running tests again doesn't really help sensitivity, for that you would
most likely have to use some chemical process that amplifies the effect of the
contaminants you are looking for. Hence the problem of having to know what to
lool for in validating new chemicals, if you need ultra pure chemicals.

------
maxander
Every once in awhile, one hears about some accident like this one at a single,
physical factory, followed by a laundry list of tech companies that are
adversely effected. It powerfully demonstrates how awfully centralized
microchip production is; individual plants can account for substantial
fractions (or even majorities) of entire product categories, which means that
individual incidents can have outsized effects. I don't know if that sort of
risk has been fully internalized by the tech sector.

For that matter, how specialized is the market for whichever high-purity
chemicals were behind this incident? What portion of the semiconductor
industry was purchasing from the same supplier? We may not have heard the last
of this.

~~~
rburhum
You mean like AWS being down?

~~~
YjSe2GMQ
Or a bad BGP route being broadcasted, whether intentionally or not:

[https://bgpmon.net/bgp-leak-causing-internet-outages-in-
japa...](https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-
beyond/)

------
fizixer
I love how Moore's law is a complete marketing gimmick at this point:

\- Intel not only hasn't released 10nm chips yet, they even stopped
publicizing transistor count of their 14nm, the node from five years ago.

\- Not to mention their i7 chips node over node being marginal improvements
for the past 10 years or so.

\- And all of a sudden, Intel's competitors seem to be right on track with
7nm, with the Moore's law giant itself, and with most resources, is struggling
at 10nm.

------
dooglius
Do TSMC or their customers have insurance for this sort of thing? It would
seem prudent given the stakes.

------
dragosmocrii
Should we expect an increased cost as consumers? I remember when Hynix foundry
had a fire, ram prices went up a lot

~~~
sydney6
I believe so, yes. Supply and Demand.

------
IMTDb
This does not bode well as well for Nvidia who already had a hard Q4 2018.
They expecs to report $2.2 billion in fourth quarter revenue versus the
previous guidance of $2.7 billion.

~~~
burk96
I know it's wrong of me, but I can't help but be a bit relieved with the
recent bad news for Nvidia. This time last year (and the year before that) it
was looking like they were about to have a total monopoly on the GPU market.
This will give AMD time to catch up a bit and Intel a good market to release
their own GPUs.

With Intel's CPU dominance being withered by AMD and ARM solutions, and
Nvidia's own dominance being thrown into question, the hardware industry is
getting more interesting all the time.

~~~
tass
I like seeing underdogs create something better, but a failure like this just
holds everything back.

Prices will be higher, and the need to compete on quality/performance will be
lower.

------
gjsman-1000
Oh, that is absolutely disastrous if true. With their tight margins... oof.
What can I say? That is ridiculously bad.

~~~
ChuckMcM
Their margins aren't as tight as you might expect. Their last quarter
report[1] puts their gross margin at nearly 48% and their operating margin at
37%. That is pretty healthy and suggests they can easily withstand this sort
of event.

[1]
[https://www.tsmc.com/english/investorRelations/quarterly_res...](https://www.tsmc.com/english/investorRelations/quarterly_results.htm)

------
ggm
Primo Levi "the periodic table" has a couple of chemist stories from his post
WWII experiences, which go to this problem. unknown adulterants, effects on
process chemistry, the wierd moments in "DONT CHANGE IT" culture. Older times,
but similer problem? And, beautifully written.

------
slenk
What are "wafers"?

~~~
dragontamer
After a Silicon Crystal with 99.99999999+% purity is grown, they are sliced
into large pieces called "Wafers". Transistors are then built on top of the
crystal structure. All together, these transistors create chips. Somewhere
between 75 to 7500 chips are made per wafer, depending on the size of the
design.

Yes, 10+ 9 purity. And the impurities in those crystals were put in there on
purpose (to make N-type or P-type Silicon). I never personally understood the
chemistry or physics behind the process, but its always cool thinking about
how exceptionally pure this whole process is.

In any case, if a run of wafers is bad, that can easily be hundreds-of-
thousands of dollars worth of chips per wafer. For whatever reason, it seems
like these wafers did not have the 99.99999999% purity needed to successfully
make chips, so everyone's chips are ruined. Whatever the issue is, you can
rest assured that a ton of business folk are going to be pissed.

~~~
NotAnEconomist
> This accident stems from the fact that imported chemical materials do not
> meet the requirements, resulting in flaws in the wafers produced.

It's possible one of the washes or doping agents was contaminated, rather than
the wafers themselves.

------
olliej
Oof, I hope they can absorb the cost :-/

~~~
extrapickles
Since it sounds like they didn’t catch it until near the end, it’s probably
north of $65k/wafer. Probably single digit billions would be an accurate guess
for the total loss to TSMC.

~~~
rubbingalcohol
Worth mentioning that this is TSMC's fuckup, not Nvidia or the other companies
using their fab.

This is so incredible it makes me wonder if we're looking at industrial
sabotage. I can't believe TSMC wouldn't have safeguards in place to prevent
contamination under normal circumstances, much less at this scale.

~~~
jannes
Who benefits from this, though? The first one that comes to mind is Intel...
Although I am not suggesting that they did it. It's just a hypothetical
question.

~~~
zamalek
Anyone that doesn't use the Nanke 14 factory specifically (or uses the factory
only for some of their products).

~~~
JudasGoat
Or perhaps even a vendor that had contracted more capacity(in Nanke 14) than
the present market demands, could benefit from a delay in production.

------
libertine
This could be good for AMD.

~~~
SteveNuts
I thought AMD was using TSMC now?

~~~
nonbel
This is about TSMC's "16/12nm process".

AMD is using TSMC's 7nm process and Global Foundries' 12/14nm processes:
[https://www.extremetech.com/computing/276169-amd-moves-
all-7...](https://www.extremetech.com/computing/276169-amd-moves-all-7nm-cpu-
gpu-production-to-tsmc)

