
Intel wants to kill the traditional server rack with 100Gbps links - orrsella
http://arstechnica.com/information-technology/2013/04/intel-wants-to-kill-the-traditional-server-rack-with-100gbps-links/
======
zdw
They have the parts to do this:

\- High speed copper physical layers are cheap - Thunderbolt today has 40Gbps
PHYs, and Infiniband/10GBE both extremely inexpensive compared to a few years
ago.

\- Their chips are getting smaller and more power efficient, and the Atom
finally has an ECC part.

\- As clockspeed is not longer hockeysticking up, people are transitioning in
droves to multi-system parallel task model, which favors lots of medium speed,
efficient cores.

In short, in the near future general purpose computing is going to look more
like HPC clusters, because the software side of things has, in most cases,
caught up.

~~~
Retric
Latency is a significant issue with this approach. A SSD doing 100,000 IOPS
per second that's 3 feet from the CPU is noticeably faster than one that's 100
feet from the CPU.

~~~
uvdiv
Since the speed-of-light delay for 100 feet is 100 ns, and SSD latencies are
on the order of 10-100 us, this shouldn't be a fundamental issue. Maybe it's
caused by the latency to the disk cache (DRAM)? In which case perhaps you
could separate that from the disk, and move it closer to the CPU.

~~~
bradleyjg
At 2Ghz the speed of light in a vacuum delay is a clock cycle every 15cm.
Current fiber and copper transmission is about 70% of c, so that's a clock
cycle every 10.5 cm.

A few clock cycle might not matter for bulk storage, but Intel is also talking
about separating main memory from individual processors. There individual
clock cycles do matter. Witness the rise of low latency premium RAM.

~~~
geon
We are already doing that, in many steps.

------
swalsh
I know nothing about hardware, but i'm curious. Isn't a whole lot of the speed
we're getting out of systems these days a ramification of putting everything
on the same piece of silicon? If each component is physically separated, isn't
that going to impact latency?

~~~
wting
Think of the cache pyramid.[0] Latency may be higher, but if there is enough
work / data then it's still efficient to do work remotely / put data farther
away.

From what I understand, the article is about two things:

1\. Hot swappable components.

Oversimplified, but imagine being able to add / remove CPU and RAM to your
server like you can with disk space and USB thumb drives.

2\. Shareable resources / load balancing.

If one server is using all its CPU and another is using all its RAM, they can
now use each other's resources.

You know how AWS is on-demand, scalable computing? This is the same thing but
at a hardware level with CPU, RAM, disk, and network resources.

[0]: <http://static.ddmcdn.com/gif/computer-memory-pyramid.gif>

------
jbooth
Even without crazy fabric setups, how have we been stagnating at 1GbE for like
10+ years now? Shouldn't 10, with a normal ports-and-switches type of setup
become normal someday soon?

~~~
lmm
I think mainly there's not the demand for it - you'd need a fast SSD to
saturate 1GbE, and nothing remotely consumer-level can saturate 10GbE yet.

~~~
mason55
Who says it has to come from hard drives? Look at things like redis or
memcached that are serving out of RAM.

~~~
noselasd
But (ordinary) people don't run redis on their home network. 10Gb Ethernet is
very common on servers in datacenters today, but it hasn't yet caught on to
the consumer market.

------
jtbigwoo
This sure looks familiar. Intel wants to build mainframes, basically.

~~~
zmonkeyz
I came in to say this and didn't want to get all the heat. Thanks for stepping
up. :)

~~~
jtbigwoo
Turns out it was a popular sentiment! :)

To be clear, I like mainframes. They make certain problems much easier. I
worked in the company that maintains the world's largest DB2 installation and
it was amazing and terrifying.

------
HeXetic
Isn't this sort of like turning an entire rack into just a big blade server
enclosure?

~~~
dsr_
Yes, actually.

Instead of a dumb rack -- four posts and screw holes in the right places --
you get a blade chassis 72U high. Centralized but redundant power supplies
with integrated monitoring and per-feed software control. A built-in KVM.
Large slow(er) fans that push more air more efficiently and more quietly, or
perhaps a liquid cooling system that provides a clean, standardized disconnect
for each component. Assignable resources -- instead of running virtual
machines, you run re-configurable real machines, where you start by selecting
a number of processors with associated RAM and add in storage.

On the one hand, this will have less in common with consumer hardware, so
economies of scale will not be shared. On the other hand, server hardware is
already substantially different from high-end consumer hardware, so it's
probably not that big a deal.

~~~
samstave
Heh - I was working with a MIPS processor designer on this exact model of
device in 1999 - we basically just sketched out the idea and talked about what
it would take to have a fabric rack with a standard set of interconnects so
that various vendors could build cards to go into it.

We discussed the challenges of various signaling that would prevent companies
from being willing to participate. But this idea is really old.

------
amalag
Sounds like Intel is validating AMD's acquisition of Seamicro and their
"Fabric Compute Systems"

<http://www.seamicro.com/>

~~~
zurn
Yep, tempted to call it a Seamicro clone (if they build it some day).

------
cramerica
[http://en.wikipedia.org/wiki/Mainframe_computer#Characterist...](http://en.wikipedia.org/wiki/Mainframe_computer#Characteristics)

------
lifeisstillgood
This does not sit so well on a front page with a "fusion drive to Mars" link.
This story has almost as much if and when about it - developing new silicon
photonics, a reference architecture in 2014. Sounds a lot like a intel
territorial fight than an actual product.

Probably a nice idea though

------
grok2
Whenever a high-profile company gets an article in a prominent publication
touting a "new" feature/hardware/system that really is already in use in the
industry for years, then you know that feature/hardware/system has arrived.
What they announced is just a different spin on pre-existing technology.

Just like the way Cisco, et. al. are co-opting
SDN...[http://www.lightreading.com/blog/software-defined-
networking...](http://www.lightreading.com/blog/software-defined-
networking/cisco-sees-daylight-for-sdn/240152079?queryText=opendaylight)

------
njharman
Is it just me or does this seem very much like mainframe architecture from 30
years ago? Not that 30yr design is bad. More that all trends/technology are
cycles.

------
johngalt
Seems like there are two architectures that are commonly adopted. One is to
make the server ever larger and more modular. E.G. separating
processors/memory and storage as separate entities that are individually
configurable (like this article). The other is to make the server smaller and
more parallel. E.G. 1000s of single socket/single hdd servers running in
aggregate.

------
karlkatzke
I see this flying about as well as blades do. There's reasons to use blades,
but a 42U stack of pizza boxes with a commodity interconnect (even 10gbps) is
almost always cheaper than a single blade center.

------
6ren
One of Intel's advantages is being a process node ahead of other foundries:
22nm vs 32nm.

------
papsosouid
This already exists. This already existed in the 90s, and was in fact very
normal. Sun, DEC/compaq, HP, IBM, etc all sold systems like this. Why does
intel always pretend doing something 20 years late is innovation?

~~~
ajross
That's true, but really only in a specious way. If you want to argue that sort
of logic, then people were building machines "like this" back in the 60's when
the "rack" was a big card cage into which you could plug (sometimes) arbitrary
amounts of application-specific CPU/memory/storage/IO resources.

But what's actually happening is that Intel is building a fabric interconnect
which can serve data at DRAM-bandwidth-or-higher speeds. And that's certainly
something that hasn't been done in the modern world. The headline might lead
you to believe it's the architectural idea that's the new thing here, but it's
really the pure technology that's the interesting bit.

~~~
papsosouid
>But what's actually happening is that Intel is building a fabric interconnect
which can serve data at DRAM-bandwidth-or-higher speeds

Yes, that is exactly what I am talking about. It was perfectly normal to
purchase systems in the 90s that worked that way. A cabinet powered by a
single power node, and you plugged in CPU nodes, memory nodes, I/O nodes, etc.

<http://en.wikipedia.org/wiki/DEC_7000/10000_AXP>

~~~
ajross
Right. And yet people _stopped doing that_ when 512 bit wide cache lines
pulling at 30-60MHz became the norm. The interconnect just wasn't fast enough,
and everything moved onto a single board and stayed there. For almost 20 years
the best we've been able to do are things like Infiniband that are far slower
than on-board DRAM.

Now (apparently) there's a new interconnect that can do the job. That's news,
not "same old boring stuff".

~~~
zurn
> For almost 20 years the best we've been able to do are things like
> Infiniband that are far slower than on-board DRAM

The SGI shared-memory big iron machines (Origin & Altix) are more recent and
can have memory-only nodes. The most recent NUMA Altix was launched in 2009,
I'm not sure if the later machines managed to keep the Origin 2k era goal of
having as much remote memory bandwidth as local.

<http://en.wikipedia.org/wiki/NUMAlink>

