
Point to Point in the Datacenter with Andy Bechtolsheim - rbanffy
https://www.nextplatform.com/2019/10/10/to-point-in-the-datacenter-with-andy-bechtolsheim/
======
zamadatix
I'm on the enterprise networking side and I find it funny that a lot of the
time I have to explain that it would be cheaper to buy higher speed links on
the new server regardless how little bandwidth is actually going to go through
them. I've hard a hard time using 32 port 100G switches in a few racks (can
break out into 128 25G ports) instead of 48 port 25G switches (same ASIC
family/generation, basically the same price per switch) because a lot of
servers being ordered "don't need 10G NICs" and the 100G platforms don't
support 1G anymore.

~~~
DiabloD3
That's also one of the hats I wear at work.

48 port 10GBE switches that support 1, and have 40GBE uplinks are pretty
inexpensive. The real problem is, 10 isn't replacing 1, the nbase-T/802.3bz
2.5/5 NICs are (which, they themselves, are just 802.3an 10GBE ran at half or
quarter clock)... the places where 10 replaced 1 is where that happened years
and years ago, and 40 is now replacing 10. A lot of the newer 48 port 10GBE
switches have 802.3bz support, and a lot of newer platforms are coming out
with 802.3bz by default in 2020 or 2021.

You're right with the rest, though, the 10gbit serdes family (4x10=40,
10x10=100) is being replaced with 25gbit serdes to build 25, 50, 100, and 400;
and even more ridiculous is 100gbit serdes is coming out soon, so theres going
to be 8x100=800 as a future 802.3 spec.

~~~
zamadatix
I'm curious if 3bz is ever actually going to take off in force. The 11ac wave
2 marketing campaigns certainly pushed it but in the end it never actually
mattered since goodput was never actually that high, the same seems to be on
repeat with 11ax. Intel's 10th gen core line has native support for it so
maybe we'll start to see it trickle in through that. The biggest problem with
3bz is it's not like when 10G took over the DC where they could go phyless and
drop legacy to cut cost, all of broadcoms upcoming edge focused ASICs are
committed to supporting 10 and half duplex still after the fiasco that was the
last time they tried to remove support for those. As a result 3bz will likely
be more expensive and gig to the user will likely remain "good enough" until
they need workstation class and they'd rather have 10G anyways.

I actually got to play with one of the early 32x400G switches recently :D. I
don't think we'll be going for them anytime soon as we've only just started
rolling out 25G at scale but it certainly was something to behold.

