
Introducing “6-pack”: the first open hardware modular switch - m0nastic
https://code.facebook.com/posts/717010588413497/introducing-6-pack-the-first-open-hardware-modular-switch/
======
btoptical
These big switch boxes typically end up being about thermal management and
this box looks like thermal design was an afterthought. Also I'm not sure the
power entry design is really all that smart. Generally speaking if you're in
the market for a 640Gb/s switch (or 3.8Tb/s switch), does your data center
really not have access to 48V power? The AC/DC conversion wastes power and
space.

The QSFP's are "spaced for optimal airflow." However this spacing seems to
neglect cooling the QSFPs themselves. Belly-to-belly mounting of modules is
usually the most thermally challenging way to arrange them. The heat
dissipated by the QSFP's is generally directed towards the top of the module.
By placing open air channels between modules, they have effectively ensured
that little to no air flows over the QSFP heat sinks (which is not shown). So
there is probably a limitation on which reach codes are supported. My guess is
that because of the thermal limitations of this design, it's not truly non-
blocking in all reach configurations.

~~~
wmf
_does your data center really not have access to 48V power?_

No, they really don't. I've never heard of 48V outside of telco.

~~~
btoptical
I've been to lots of them all over the world, names you'd recognize, and
they've all had 48V.

------
rsync
I very much want to transition to open hardware models for networking gear -
for both my own home use and for rsync.net.

However, at no single site (especially my home) do I need a 6U chassis full of
switch ports.

Is there a 1U version of this on the horizon ?

~~~
kijiki
Basically anything on here: [http://cumulusnetworks.com/support/linux-
hardware-compatibil...](http://cumulusnetworks.com/support/linux-hardware-
compatibility-list/)

Some easy ways to buy are: [https://bm-switch.com/](https://bm-switch.com/)
[http://whiteboxswitch.com/](http://whiteboxswitch.com/)
[http://ipfabrics.net/](http://ipfabrics.net/)
[http://www.penguincomputing.com/products/network-
switches/](http://www.penguincomputing.com/products/network-switches/)

\- nolan co-founder/CTO Cumulus

~~~
tiernano
Are there any that are, say, 48 port GigE? 10G and 40G are a bit much for home
use...

~~~
skuhn
There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and
$700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain
support in order to continue using the product (legally). I can't imagine
getting enough utility from a switch for this to make sense for home use.

It is substantially better pricing than list for comparable Juniper / Cisco /
etc. equipment. However, Cumulus has no truly low end 1 gig switch (single
power supply, limited L3 capability), and you can absolutely negotiate Juniper
/ Cisco / etc. down to be close to or even below the pricing of the Cumulus
solution. That is tougher for 10 gig or 40 gig equipment, which is where the
value proposition kicks in for Cumulus.

Maybe Cumulus's approach is enough of a value add for it to make sense to pay
a premium, but everyone I talk to is interested in cost savings first and
better manageability a distant second. You'll still have to have Juniper /
Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they
don't have a full range of switch models.

~~~
kijiki
Stay tuned for low-end 1G platforms running Cumulus Linux.

\- nolan

------
korethr
This is rather cool, and I'm glad to see some work being done in the
networking gear space utilizing open designs and firmware like this.
Unfortunately, unless I find myself needing to network a datacenter in the
near future, it's not immediately useful to me.

That said, I do hope other developments in network gear that will be useful in
other markets emerge from this effort.

From my perspective, there's a gap in network gear between the unmanaged, low
port-count switches in plastic enclosures, targeted to home and small office
consumers, and the lower tiers of Cisco's catalog, targeted toward top-of-rack
or wiring-closet-of-a-larger-building type uses. I would love to see a managed
switch with say, 8-24 ports, supporting features such 802.11Q VLANs. I would
love to be able segment my network at home so different devices with different
performance and security needs aren't all stepping on each other's toes. And
I'd like the firmware and hardware designs to be open source, so it can
readily patched when bugs are found, and easily adapted to new use cases.

I realize that I'm an outlier and that my needs are not common, or there'd
probably be equipment on the market that met them. But it is my hope that as a
result of Facebook's work here, and similar efforts, that building such a
device will become feasible.

~~~
gh02t
I also wanted something in the category you're describing and after shopping
around a bit I stumbled across Mikrotik's products. I bought one of their 24
port smart switches and it was exactly what I was looking for. Not a lot more
expensive than some of the nicer consumer-targeted gear, but it has _way_ more
functionality.

[http://routerboard.com/CRS125-24G-1S-IN](http://routerboard.com/CRS125-24G-1S-IN)

~~~
e12e
Looks like a very nice product (along with most of the other products). It's a
little disappointing that they seem to do only the very minimum wrt the GPL:

[http://www.mikrotik.com/downloadterms.html](http://www.mikrotik.com/downloadterms.html)

I mean, sure, asking for 45 USD for a CD with the source is technically
complying with the GPL -- but it does seem a bit strange in this day and age.
Not to mention that for the source code to be useful, one would hope one could
build a working routerOS image from it -- and it doesn't appear that the CD
will enable a user to build a running image -- and therefore not facilitate
changing the product.

But apart from that, my first thought was -- can this thing run _BSD --
because pf is quite a bit friendlier than iptables (even if the latter has
gotten a lot better lately).

Does anyone know if there are any recommended alternatives to soekris for
running a _bsd switch/router (preferably running at ~gigabit speeds) ?

~~~
nl
Lots of PFSense people seem to think the PC Engines APU[1] is ok. Netgate
sells a DIY kit[2].

It's only a 3 port system though.

[1] [http://www.pcengines.ch/apu.htm](http://www.pcengines.ch/apu.htm)

[2] [http://store.netgate.com/kit-APU1C4.aspx](http://store.netgate.com/kit-
APU1C4.aspx)

------
pdkl95
Wow. That switching fabric is so much smaller than the HIPPI crossbar
backplanes we used in the 100baseT/pre-GigE switches that I used to write
firmware for... almost 20 years ago now. Of course, complexity means higher
cost, so it's obvious why nobody bought our stuff. Hopefully, this open
hardware project will have better luck!

/* it's easy to forget just how many iterations of Moore's Law have happened
since the mid-90s */

------
sudioStudio64
The work going into whitebox switching and routing is super exciting. I really
dig this stuff.

------
rustyconover
I get the feeling this is a bit like Bugatti building the Veyron, its super
cool and impressive but the vast majority of us will never be able to or have
a need to drive it.

It is still fun watch this switch go around the proverbial track, but I'm
happy knowing that I'll never have to configure, build and test a switch of
this complexity unless I really absolutely have to, with my largest caveat
being AWS disappearing from the face of the earth.

~~~
AceJohnny2
You're not the target market. You shouldn't be making an analogy with a
Veyron, which remains a luxury car for an individual. You should rather be
comparing it with a firetruck, a vehicle custom-built for a purpose you as an
individual you will never need.

I currently work in the network hardware industry (think Cisco, Juniper, etc).
Our boxes sell in the 6-figure price range, each [1]. We sell to your ISP,
wireless carrier, datacenter constructor. We're the competition this kind of
box is aiming at.

[1] it's highly specific hardware and software for a low-volume market.
Individual _chips_ used in the hardware can cost multiple thousands of dollars
each.

~~~
gonzo
> Individual chips used in the hardware can cost multiple thousands of dollars
> each.

and 10 years ago, the compute power of a QC i7 would have tens of thousands of
dollars.

~~~
AceJohnny2
Right, but we don't care about past chips. Moore's law applies to the whole
market at the same time, so today's special network chips are still much
better at what they do than today's general-purpose chips.

It's kind of stupid to say that today's general-purpose hardware can do what
yesterday's special hardware could do at a faction of the cost, because the
specialist market has also moved on and wants the performance of today's
special hardware.

------
gdooraccess
Does anyone know which team in FB is building software for these products? I
did glance FB company info but could not get the information.

~~~
aristus
I no longer work there but I can ask the usual suspects. Email is in my
profile.

~~~
gdooraccess
Thanks and sent you an email. For anybody who is looking for similar details
on the team who are behind these products, just FYI - there is not much info
on FB site or on the internet. If I get any info later, will report back.

------
arca_vorago
I just want a fully open source switch that's ready for enterprise, from the
h/w to s/w. Linux preferred but bsd based is acceptable.

I've been a "cisco guy" since 2001 or so, but I am so tired of them. Licensing
fees kill budgets that could be used on other things, and you end up
surrounded by _consultants_ that only ever touch Lozedoze systems insisting
that "nobody ever got fired for buying Cisco". Smartnet is a must for some
equipment, yes, but I'm so ready for a paradigm shift in networking.

I've really been watching ubiquiti and their switching/routing products, they
seem very promising but not quite prod ready. I am impressed with Dells open
switches too.

Edit: The Microtik stuff being linked elsewhere here is looking pretty awesome
too. Not quite FOSS but still.

------
charlesnw
So this is very interesting.

As someone who runs a 6509 (almost fully populated with line cards and
specialized controllers) as my HOME core switch (with a second one on the way
for full redundancy), an open modular switch is very cool.

(I'm currently hacking on OpenFlow -> NETCONF bridging, to bring typical SDN
capabilities to legacy Cisco environments). Hence my rather..... extensive
home network.

This is very similar to what Cisco is doing with Nexus and the "fabric
extender" TOR replacement kit. I'll have to see if Facebook has any of this
stuff in GIT and stand it up in a VM environment and play with it (I already
do a bunch of OpenFlow stuff on OpenWRT and am looking at implementing an open
southbound API on FPGA on the parallela board).

------
VLM
I was expecting a link to something like this, but more advanced

[http://opencores.org/project,mac_layer_switch](http://opencores.org/project,mac_layer_switch)

and some links to digilent dev boards for hardware, however the linked project
is a bit more ambitious, all full of custom ASICs and such.

An open source switch made out of COTS FPGA dev boards would be interesting.
So you'd use something like

[http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,400...](http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,400,1228&Prod=NETFPGA-1G-CML)

But what to use as a COTS "just unpack it all, plug it all together, upload
the firmware and go" backplane?

~~~
virtuallynathan
That NIC seems quite overpriced for 4x1G...

~~~
wmf
It's not a NIC; it's an FPGA that can be programmed to do things that NICs
have never dreamed of.

~~~
virtuallynathan
I understand that, but you can get more powerful ones for not much more money.

~~~
detaro
NICs or FPGA boards with NICs? If the latter, I'm all ears

~~~
virtuallynathan
Netronome, [http://www.mellanox.com/related-
docs/prod_adapter_cards/PB_P...](http://www.mellanox.com/related-
docs/prod_adapter_cards/PB_Programmable_ConnectX-3_Pro_%20Card_VPI.pdf), there
are others for more $$ using the Altera Arria 10, or other Xilinx FPGAs with
2x40G or even 4x100G.

------
fidotron
Isn't this just wrapping for the COTS switching ASICs or am I missing
something big?

Almost all "open" projects seem to hit a point where the line between open and
closed is quite arbitrary, even if they choose not to see it that way.

~~~
wmf
"Wrapping commodity ASICs" is worth $4B:
[http://www.arista.com/](http://www.arista.com/)

If Facebook wants to give me that value for free I won't complain.

~~~
SEJeff
FWIW, all Arista hardware is AWESOME. They run a lightly modified very minimal
linux operating systems that is very clearly (not even really hidden) Fedora.
I got the privilege / opportunity to reverse engineer one of these to see if
it was hackable (ie: add custom hardware into them or run apps directly on the
switches).

Pretty much the entirety of the Arista userspace is python with all of the
magic really in their ASICS. Truly, I've not seen many better engineered
pieces of equipment. At the time (I did this > 4 years ago), the Arista
switches had dual core AMD Athlons with a few G of ram. They were willing to
put more RAM in one, or let us upgrade it. For a so called "vendor product"
Arista clearly gets it. They have some really solid Linux guys working for
them. When you get the pleasure of using their python apis or pull one apart,
it shows that they know what they're doing. It isn't completely insane to
think of running apps directly on the switches when they are simply x86_64
Fedora boxes with some fancy asics and a lot of interfaces.

------
MrZipf
This looks great. They also have the advantage that James Hamilton mentioned
AMZ got from building their own hardware - the get to do a cleansheet s/w
design and not have to support all the gazillion options that Cisco does.

The talk was on HN a little while ago, worth watching:
[https://www.youtube.com/watch?v=JIQETrFC_SQ](https://www.youtube.com/watch?v=JIQETrFC_SQ)

The scale at FB means they're probably saving hundreds of millions doing this.

------
aaronsnoswell
Am I the only one who read the title and thought it referred to a 'pushbutton'
style electronics hardware switch? I would have been quite excited to see
that!

------
cordite
I wonder if youtube will ever notice that an uploaded video is quiet and only
on one channel, and allow for normalized mono listening..

