
Goodbye, Motherboard. Hello, Silicon-Interconnect Fabric - craigjb
https://spectrum.ieee.org/computing/hardware/goodbye-motherboard-hello-siliconinterconnect-fabric
======
craigjb
It's been fun to see Dr. Subu present this concept and prototypes at several
conferences, and the level of integration possible is absolutely insane. I
think the industry is definitely moving toward chiplets, such as the latest
AMD release.

I definitely think we will see more chiplets and more standardization on
interfaces between chiplets. The focus will be on how to minimize energy per
bit transferred (a big topic in Subu's talks) and how to minimize the die area
used for inter-chiplet communication. In monolithic silicon, you don't have to
think about die area, since your parallel wires between sections might just
need a register or two along the way. With chiplets, you typically can't run
wires at that density yet, so you still have some
serialization/deserialization hardware. But, since it's not crossing multiple
high inductance solder balls and PCB traces, you can get away with less.
Hopefully also you can get away without area-intensive resynchronization,
PLLS, etc.

I think it will definitely be awhile before this kind of integration is used
outside of niche cases though. The costs are just insane. You have to pre-test
all manufactured chiplets before integration, and that test engineering is
nothing to sneeze at. If you don't then you have all kinds of commercials
issues about who is liable for the $500k prototype one bad chip broke.

On the bright side, I see the chiplet approach benefitting other integration
technologies. For example, wafer level and panel level embedded packaging
technologies can be used for 1-2um interconnects now. You won't get a wafer
sized system out of it with any kind of yield, but it's probably the direction
mobile chips and wearables will go.

Anyway, disorganized info-dump over.

~~~
jabl
I agree this looks promising, though I'm not an expert in this field.

But the title is a bit, well, overpromising or broad. I don't think we'll
replace traditional motherboards anytime soon (except maybe in smartphones?).
Rather, it will be an incremental progress.

\- first, SoC's will be replaced with chiplets

\- then we'll start seeing more and more stuff being integrated on this wafer.

\- say, instead of a server motherboard with multiple sockets, have all the
CPU chiplets on the same wafer and enjoy much better bandwidth than you get
with a PCB

\- integrate DRAM on the wafer. This will be painful as we're used to being
able to simply add DIMM's, but the upside is massively higher bandwidth.

The motherboard pcb per se will live for a long time still, if nothing else
then as the place to mount all the external connectors (network, display,
pcie, usb, power, whatnot).

~~~
elihu
> integrate DRAM on the wafer. This will be painful as we're used to being
> able to simply add DIMM's, but the upside is massively higher bandwidth.

One way I imagine this working out is that, instead of just replacing the
plastic motherboard with a silicon motherboard, you eventually do away with a
single monolithic motherboard entirely. Instead, you have "compute blocks"
(comprised of chiplets bonded to a silicon chip, or conventional chips on a
conventional circuitboard) that connect with each other via copper or fiber
optic point-to-point communication cables, and you can just wire them together
arbitrarily to build a complete computer. Like, you might have a couple blocks
that house CPUs, one or two that have memory controllers and DRAM, and maybe
one with a PCI bus so you can connect peripherals, and you can connect them
all in a ring bus. You could house these blocks in a case and call it a
server, or connect a lot more blocks and call it a cluster.

The main advantage of such a setup is that you don't have a single component
(the motherboard) that determines how much memory, how many processors, or
what sort of peripherals you can have.

~~~
smrq
I've been playing a lot of TIS-100 lately, so this architecture sounds like a
nightmare to develop on.

~~~
Doxin
You know, I bet the TIS-100 wouldn't even be all that tricky given a) way more
cells and b) a compiler to abstract a bunch of stuff.

------
bArray
Yeah... I'm not entirely convinced about this future.

* PCBs are cheaper to manufacturer than silicon wafers.

* PCBs can be arbitrarily created and adjusted with little overhead cost (time and money).

* PCBs can be re-worked if a small hardware fault(s) is found.

* PCBs can carry large amount of power.

* PCBs can help absorb heat away from some components.

* PCBs have a small amount of flexibility, allowing them to absorb shock much easier.

* PCBs can be cut in such a way as to allow for mounting holes or be in relatively arbitrary shapes.

* PCBs can be designed to protect some components from static damage.

What I can see on the other hand is some packages end up being dropped into
the PCB and soldered at the sides. Sometimes this is done with large through-
hole capacitors, where the legs are bent and the capacitor sits in the middle
of the PCB (inside a cut hole). Other than ball packages, you could could
probably drop the majority into the PCB itself.

The other obvious option for manufacturers will be to put more tech on a
single die, but then other problems are also raised. For example, some parts
are binned based on their tested results.

~~~
userbinator
_What I can see on the other hand is some packages end up being dropped into
the PCB and soldered at the sides._

That's been done for at least 30 years:

[https://www.keesvandersanden.nl/calculators/hp32sii_repair.p...](https://www.keesvandersanden.nl/calculators/hp32sii_repair.php)

~~~
bArray
That's amazing and I'm glad that it's possible, even by the standards of older
technology. I certainly think dropping packages into the PCB is the lowest
hanging fruit for reducing depth.

------
jpm_sd
"Power turned out to be a major constraint. At a chip’s standard 1-volt
supply, the wafer’s narrow wiring would consume a full 2 kilowatts. Instead,
we chose to up the supply voltage to 12 V, reducing the amount of current
needed and therefore the power consumed. That solution required spreading
voltage regulators and signal-conditioning capacitors all around the wafer,
taking up space that might have gone to more GPU modules."

Uh, this seems like a pretty serious downside.

~~~
jjoonathan
Also: shifting PCB prototype costs in the direction of silicon is going to be
a tough sell for many applications.

> Another drawback of SoCs is their high one-time design and manufacturing
> costs, such as the US $2 million or more for the photolithography masks

> ... 6 paragraphs later ...

> Patterning wafer-scale Si-IF may require innovations in “maskless”
> lithography.

------
seltzered_
“Silicon-interconnect fabric, or Si-IF, offers an added bonus. It’s an
excellent path toward the dissolution of the f(relatively) big, complicated,
and difficult-to-manufacture systems-on-chips that currently run everything
from smartphones to supercomputers. In place of SoCs, system designers could
use a conglomeration of smaller, simpler-to-design, and easier-to-manufacture
chiplets tightly interconnected on an Si-IF.”

Reading this reminded me of a remark from Bunnie Huang’s teardown of a dirt
cheap ‘gongkai’ cellphone
([https://www.bunniestudios.com/blog/?p=4297](https://www.bunniestudios.com/blog/?p=4297)):
“To our surprise, this $3 chip didn’t contain a single IC, but rather, it’s a
set of at least 4 chips, possibly 5, integrated into a single multi-chip
module (MCM) containing hundreds of wire bonds. I remember back when the
Pentium Pro’s dual-die package came out. That sparked arguments over yielded
costs of MCMs versus using a single bigger die [...] I also remember at the
time, Krste Asanović, then a professor at the MIT AI Lab now at Berkeley, told
me that the future wouldn’t be system on a chip, but rather “system mostly on
a chip”. The root of his claim is that the economics of adding in mask layers
to merge DRAM, FLASH, Analog, RF, and Digital into a single process wasn’t
favorable, and instead it would be cheaper and easier to bond multiple die
together into a single package. It’s a race between the yield and cost impact
(both per-unit and NRE) of adding more process steps in the semiconductor fab,
vs. the yield impact (and relative reworkability and lower NRE cost) of
assembling modules. Single-chip SoCs was the zeitgeist at the time (and still
kind of is), so it’s interesting to see a significant datapoint validating
Krste’s insight.”

I wonder if there’s any advantages to si-if from an ewaste (aka reverse
logistics, cradle-to-cradle) perspective

------
mooman219
Another drawback the article doesn't mention is tight component coupling
removes the ability to treat the components as separate pieces. This can make
repair or upgrades extremely difficult. This might be desirable for companies
against the right to repair; if it's illegal to use software to prevent
repair, making it sufficiently difficult by hardware design is a possibility.

~~~
repair_man
Is anybody actually repairing a faulty motherboard? Let's one of the chips or
some resistor is broken.

~~~
zeta0134
Yes, quite often in fact. On older motherboards, often the only problem is
something benign like a blown capacitor, which is a few cents at the hardware
store and a few minutes labor to desolder the old and attach the new. It's a
really common part to wear out due to heat stress, and it manifests as funky
logic problems, since the caps are mostly there to clean up the line noise.
I've saved a few flatscreen monitors this way, that were flickering and
unusable; I'll take a $5 handful of parts over a new $179 monitor any day of
the week.

~~~
ryanwaggoner
I have only a rudimentary understanding of electronics and what the various
components do, so this may be a dumb question, but how on earth do you
diagnose an issue like this? If my monitor dies or my printer stops working
and I open it up, I’m completely bewildered at what steps you’d take to figure
out that some random resistor or capacitor out of (what seems like) thousands
of components is the one causing the issue. Do you just go through and test
each one with a multimeter until you find it?

~~~
bradstewart
If a consumer electronic device suddenly refuses to power on, it's a failed
electrolytic capacitor most the time (in my experience). Finding them requires
effectively no knowledge of circuits. Sometimes one is visibly
deformed/exploded, but I usually just replace them all.

If that doesn't work, yea you pretty much start testing things with a
multimeter starting from the power source. But if it isn't a capacitor
failure, it's probably ESD or power-surge related damage and not worth trying
to fix.

~~~
jdnenej
I attempted to repair a coffee machine that had a short that left a black burn
on the PCB. Replaced the parts around it and it still wouldn't turn on. Likely
the power surge destroyed a lot of things.

~~~
pbhjpbhj
Those sorts of power faults can burn out traces on the circuit board too (you
can bypass the break sometimes), and often there are fuses that would n need
changing (in my very limited experience).

Louis Rossman on YouTube has a lot of stuff on circuit repair that I find
good,
[https://m.youtube.com/watch?v=_at9Jy3dfeY](https://m.youtube.com/watch?v=_at9Jy3dfeY).

------
ChuckMcM
This step is inevitable in my opinion. This is especially true for mixed
signal (that is some digital and some analog components) systems. It's early
fore-bearer (multi-chip modules or MCM) was instrumental in IBM getting its
later mainframes to hit the density and thermal constraints.

Like the authors, when I saw AMD's chiplet pictures for the Zen2 chips I felt
that we would see this expanded. Intel has also done some interesting optical
chip to chip interconnects that would facilitate assembling these newer
multichip modules into chassis that route signals to and from the outside
world.

The next (and perhaps last) element to fall into place is a way to efficiently
cool these systems. One of the problems that large data centers face is not
that they want "smaller" boxes but that they need to pull enough heat out of a
rack of servers in order for them to reliably function.

~~~
jabl
For data centers, while air cooling might be running out of steam, there's
plenty of leeway in water cooling. So I don't think that's an insurmountable
problem.

~~~
ChuckMcM
Having been inside Google's data centers I agree with you :-). I did a top
level, high efficiency data center design for a multi-tenant hosting with
Google scale economics as an exercise once, talked with some potential
partners who were very enthusiastic. It would cost roughly double what the
existing warehouse type data centers cost to build, but it would repay its
costs faster than they did (given colocation cost structures at the time). It
also leveraged some of the open compute designs to achieve better density.

------
mhb_eng
For reference, there are companies that have developed this to a commercially
viable level. First that comes to mind is ZGlue, a company that has the design
tools, interconnect fabric, and chiplet ecosystem required to deliver these
kinds of devices.

[https://www.zglue.com/](https://www.zglue.com/)

~~~
jpm_sd
This is really impressive work.

------
Yaa101
The downside is that independent entities are not anymore able to design
hardware projects, there are only a few companies that can design and make
such integrated circuits. Cpu's are not only made for computers but for all
sorts of appliances. Time will tell if these big companies will sell chiplets
and the tech to assemble them onto silicon to 3rd paties. PCB tech is fairly
democratic in that sense.

~~~
jabl
I think it's more fruitful to (initially at least) see this as an alternative
to single chip SoC's rather than an alternative to PCB's.

------
korethr
I kinda like this idea, but I think it only makes sense where the end product
sold to the intended consumer is wholly integrated. So, this might make sense
for a smartphone. Or perhaps custom systems in Google- or Amazon- scale
datacenters, with a number of different custom types tuned to exactly the work
each is intended to do, and packed as densely as the intersection between
thermodynamics and economics will allow.

But, I don't think we can escape the need for packages and PCBs entirely. At
some point, you're going to need to interface with something that either A)
you don't or can't control, or B) something at appropriate scale for
interfacing with the humans whom the fancy system is ultimately supposed to
serve. In either case, here come standard connections that are much bigger
than the chiplet dies or the interconnect fabric between them, and thus, the
need for PCBs to connect the Systems-on-a-wafer to the outside world.

As such, I think it will be a while before I can pick components out on
Mouser's website, and have all those component chiplets fused to an
interconnect wafer and delivered to my home or employer's shipping dock
(though that would be damned awesome).

~~~
gumby
> As such, I think it will be a while before I can pick components out on
> Mouser's website, and have all those component chiplets fused to an
> interconnect wafer and delivered to my home or employer's shipping dock
> (though that would be damned awesome).

I agree. People have been talking about that future since at least the 1990s
(back then SOC didn't mean a somewhat standard chip with a lot of peripherals,
but literally a custom die with your own hardware on it). "Hardware/software
co-design" was part of the jargon of the day. I'm not holding my breath.

But I do imagine seeing a lot of consumer products go this way. Not a $3 IoT
light switch/malware vector, but anything that gets rid of connectors is a win
on both the BOM and reliability standpoint, but anything in the $50-$500 price
range with volume over a million units is probably worth it.

------
cududa
Color me skeptical, but I remember being 12 seeing this on The Screen Savers
on TechTV in 2003

------
holy_city
I like the idea of faster and smaller devices but dislike the idea of
expensive production methods that are only realizable with heavy capital
investment and large volumes of products.

Like I can realize a PCB prototype in my apartment (or rather, my parking lot
without telling my landlord), or rent a CNC machine at a makerspace/public
library to do it. If I need the thing fabbed with a nice solder mask and silk
screen, I can have it made for < $20 domestically with under two weeks lead
time. Component sourcing is even easier.

But where do I go to have the chiplets I need for the circuit? Organize the
logistics to ship them to the clean-room where they can be packaged on this
fabric? How many widgets do I need to ship for this to be viable, or for the
contract fab to not laugh at me? How do I prototype? How long is the lead
time?

It just seems like there's a _lot_ in the way of this being viable for run of
the mill projects.

~~~
jabl
Increasing capital costs and consequent dependence on huge volumes have been a
defining feature of the semiconductor industry since the invention of the
integrated circuit.

AFAICT this technology will have no impact on the low end hobbyist end of the
market, but rather (if all works out) enables those $zillion behemoths to
produce ever faster systems since they won't be as bottlenecked by off-chip
bandwidth as they are with today's PCB's.

------
tartavull
You can order custom pcb from China for a couple of dollars. How likely is
that this technology will be affordable at small quantities?

~~~
jabl
Seems one of the main advantages is higher density of conductors and
connectors compared to a PCB. Aligning those dielets with micrometer precision
would seem to require equipment out of reach for the hobbyist. And then
bonding them, while they don't use solder but rather temperature and pressure,
again with such small connectors I guess the line between bonding and
destroying the thing is very very fine. Again requiring specialized and
presumably expensive equipment.

------
localhost
Does anyone know how this compares with AMD's Infinity Fabric? How is it
similar or different from it?

~~~
osamagirl69
Infinity fabric is a software thing (a way to distribute information over a
buss similar to pci-e) where as this is a physical piece of hardware that
electrically passes the signals between 2 chips.

I see no reason why infinity fabric could not be routed through a silicon
interconnect, but it would be wasteful. The whole point of silicon
interconnects that the hardware protocols like pci-e are no longer necessary
and can use much lower power/cheaper ways to transfer data.

------
a0zU
It seems like promising technology but the cost of making these chiplets must
outweigh their benefits.

------
wand3r
I think it should be called Silicon Fabricated Interconnect so we can just
call it Si-Fi...

------
NetOpWibby
Cyberpunk 2077, here we come!

------
BAReF00t
Gloodbye customizability and modularity, from a user standpoint.

I guess Apple will hire them pretty soon.

------
cphoover
Will this increase the number of bit flips from cosmic rays?

~~~
brootstrap
All the cool kids these days are _wearing_ nodeJS , not just writing it!

------
ColanR
It sounds like hardware side of computer science is rediscovering the UNIX
philosophy - each unit does one thing, well, with standardized interfaces.

------
Causality1
>pack dozens of servers’ worth of computing capability onto a dinner-plate-
size wafer of silicon.

Congratulations on the insurance money from your building burning to the
ground.

