From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used...
I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible.
The fact that it is a a specific interval indicates a stuck bit in memory.
Hardware designers basically started making bad decisions on this issue around the time that VLAN tagging was introduced, as well as harware forwarding of IP packets. When VLAN tags are inserted or removed, the CRC of a packet needs to be adjusted to reflect the inserted, removed and/or modified bytes from the VLAN header. Additionally Both the CRC and IP checksum of a packet needs to be adjusted when TTL is decremented as part of IP routing.
When implementing this functionality, the naive hardware designer will strip the existing CRC from the packet, modify the contents of the packet and then reuse the handy dandy CRC calculation block to place a newly calculated CRC on the packet. Similar choices are made for the adjustment of the IP/TCP/UDP checksums. If any errors are introduced in the contents of the packet by the data path prior to the new CRC is calculated, this results in the CRC being "corrected" to include the erroneous data.
A far more understanding hardware designer will instead calculate how to adjust the CRC by the changes introduced in the packet contents. Sadly, this is far more complicated to get right, and it goes against the drive of hardware designers to reuse blocks of code wherever possible. Every hardware designer working on networking has a block of Verilog or VHDL code to calculate and append a CRC to a packet. Only the most dedicated will attempt to apply only the delta needed to the CRC or checksum.
I'm not a hardware designer but I routinely deal with low level networking shenanigans and I must admit that I never considered that it would be possible to update a CRC without recomputing it fully (unless you were just appending data of course).
You can use ping to more easily hunt these types of issues, for example `ping -A -c 100 -s 1000 -p deadbeef` will show the difference if there is a flipped bit in the payload. You can generate patterns with xxd.
This kind of incident happened to me in a system that was supposed to have high availability. We had failovers for hardware, but it seems that a network device that was supposed to have HA (and was set up to pass the functionality to another device in case of failure) did not have ECC memory. One memory bit got stuck at 0 and the event was not detected at network level, as the data was repacked with a "clean" CRC. For some reason the packet headers were not affected by this, maybe because they were kept in a separate memory zone or because of memory alignment. So the device did not report any kind of suspicious activity, no errors in its statistics.
On the application side the effects were quite bad, as the data was mainly XML and, depending on where the bit was flipped, it could impact the data or the XML structure. The data had its own CRC/hash, so the packets were cleanly rejected by the application. Unfortunately the XML library from the message queue engine and the ESB we were using did not like at all when the bit flipping occurred in the XML tags (it seems fuzzing tests were not done at that point) so the message processing got stuck and we kept getting bad messages in the queues. Even worse, the queues could not be cleaned with the normal procedures because the application wanted to first display info about the messages inside - and that failed.
The network debug was non-trivial because of that header consistency - the network devices did not report any kind of packet issues, so we had to sniff the different network segments to identify the culprit. From the application point of view, we had to delete the whole message queue storage to get rid of the bad messages, and let the application handle the rest (luckily it was designed with eventual consistency and self-healing).
Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there always needs to be an escape hatch to get rid of them. Deleting the queue is too extreme.
Deleting the queues is an operational decision that I made to be able to put the system back online after the network device was replaced (the important part was the uptime/SLA). From a quick analysis of the logs the percentage of bad messages was ~90% (there was a ~50% chance that the original "touched" bit was 0 so no change was done, but the messages had multiple "touched" bits at fixed intervals).
There was an escape hatch, but the conditions to hit it were a bit complex. Implementing new message filtering of this kind at 2AM while the system was down was not feasible.
Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause
If you ping the hops with a large icmp payload, you might be able to observe the flipped bit in the echo reply. That could help isolate which hop it is.
It might be better (although harder!) to take the traceroute from example.org, instead of from the clients. Forward and reverse paths often diverge, so it's important to find the path with the error.
They need to be capturing src/dest IPs as well as ports for AT&T to have any hope of using that data.
Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop.
Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.
I think he means if everyone on AT&T experiencing the issue ran trace route to example.com some common hops would emerge, which would be a place to start investigating.
It seems like it, but it's a widespread issue across the SF/Bay Area right now, maybe wider. I've been having it for weeks and exploring it as well. I've even gone as far as ripping the certs off of the router to double check.
My guess is that they are using up to 2 HBM2 memory stacks (from the picture). Each is limited to 8GB . If they were to go to HBM2e in M2 they could get up to 2x24GB. The biggest advantage of HBM is lower power per bit as the signals are all on the package running at a lower frequency.
The memory market is getting fragmented with Nvidia having seemed to moved from HBM2 to GDDR6X (Micron being only supplier). LPDDR super low power (cell phones) and DDR4/5 for rest of market...
After more consideration it is much more likely to be LPDDR4X just like the recently released A14 chip. It would seem unlikely that they would have developed a brand new memory interface.
Flex Ethernet is that attempt to standardize the lane stripping at whatever speed. Unfortunately that effort has grown to incorporate a kitchen sink worth of other features like groups and shaping... Which raises the hardware implementation cost and lowers the adoption rate.
At the 100G/200G/400G/800G specs are highly repetitive and the only points of debate are the type of FEC required given the high speed signals, signal loss for cost effective board/cable/optics and how parallel the FEC will be (tradeoff of hardware implementation cost and added latency). FEC has been 2 engines in parallel since 400G.
The pirates are a spoof of another french Belgian comic named Redbeard (Barbe Rouge) https://en.wikipedia.org/wiki/Redbeard_(comics)
The wikipedia page even has the Asterix parodies next to the actual ones.
As one of the authors of the original paper the data is based on I have fond memories as a graduate student of standardizing the performance data to SpecInt which did not exist till the 2000s. So for the older processors that were no longer available I looked hard for processors that had both SpecInt and older benchmarks like Dhrystone and MIPS per seconds to normalize them to SpecInt performance.
The same data was used by other students of my advisor to create the Cpu database which has some more data like cache sizes and FO4.
From the latest data (https://www.karlrupp.net/2018/02/42-years-of-microprocessor-...) you can definitely see that single-threaded performance is still increasing but not at the exponential clip it was. The inflection point for that is around the mid-2000s. It is hard to eyeball because the chart is logarithmic but the rate of increase looks to be sub-exponential for the past decade or so.
MHz and Watts have clearly levelled off.
Number of logical cores looks to be on an exponential path so clearly we need to be tracking multi-threaded performance gains. Unlike your work this only needs to be backdated to the mid-2000s – multi-core ramps up when single-threaded starts ramping down – so it should be far easier.
Pretty much all applications take advantage of multi-threading now, be it multi-processing or multi-threading. Erlang is the only language I know of that has the multi-threaded paradigm baked into its core. Ruby has had Fibers since 2.2 so they are very much a tacked on idea. Perhaps what we need is to retrofit them to C, come up with an abstract machine model and language primitives that fits our multi-core world. We could call it C[].
What is really missing here is that you can build modular switches out of multiple chips that are a lot more cost effective than connecting single chip boxes mostly because the internal links in a modular chassis in connectors are a lot more cost effective than any cable or optics. This makes the Clos topology with Top or Rack using copper cables and spines out of modular switches with up to 576x100G ports in today's technology winners every time for datacenters. The supercomputing world is still stuck in requirements of extreme low latency and hence hitched to Infiniband or other specialty networks with devices with low number of ports.
Working from the 10K end-host number in the paper with a radix of 43 and 722 total router nodes, it’s the case that a given switch connects ~15 end-hosts. Fifteen downlinks to 43 uplinks seems pretty wasteful in its own right.
The numbers you cite are closer to realistic: a modified Clos fabric consisting of a four-way spine of 8-slot chassis with 36 100G blades connecting a full complement of 288 leaves, each with 4x100G up and 40x10G down (no oversubscription, at least nominally) leaves us with 11.5K hosts connected by a total of 292 switches. Even if the modular switches are 10X the cost of the ToR this is still markedly cheaper than 700+ ToR’s (and this doesn’t include actual power/cooling costs and the opportunity cost of space lost).
This also doesn’t include cabling/transceiver cost differences: 722 * 43 * 2 (62K) vs 288 * 4 * 2 (2.3K). That’s literally an order of magnitude difference.
Even if the design were approached using single-speed connections (ex: 48x10G up divided over 16 spines, 48x10G down locally) there’s still a pretty compelling numerical advantage to Clos: ~210 96x10G ToR’s and 16 16-slot chassis for 10K hosts is 226 devices and ~20K transceivers. If we assume the spines cost 10X the leaves then 370X is still almost half of 722X.
- simplified operations: can service/remove/add ToRs without affecting global routing or the forwarding capacity of the fabric.
- tor uplink flexibility: if a rack has higher bandwidth needs then double up the links (4 vs 8 in your example)
- tor location mobility: it's a lot easier to manage 4 fiber runs per tor than it is to manage 43 different fiber runs... backhaul to 4 spine blocks vs. a complex web of interconnect spreading all over the datacenter floor. with the fly network it's unlikely you can move a rack once it has been placed, at least not until you're ready to tear down the whole fabric and build something new. so you better get your rack density and layout just right. with tor you're stuck with your spine locations, but everything else can be moved around.
the fly advantages for homogeneous supercomputers built and decom'd N years later are clear... but for datacenters which grow and evolve with heterogeneous devices, fly doesn't seem to really hold up well compared to clos.
1) Copper cables only go to 7m because attenuation is too high after 7m to recover signals at 10G/25G lanes, get ready for lower length with 50G PAM4
2) Transceivers are free... no they are not fiber is one aspect of the cost but transceiver dominate
3) Missing Active Optical Cable (they are made of transceivers that do not quite meet the spec and made into a 'cable' with a fiber. These are now very important in saving costs
If you want to build a non-blocking Clos network of 2N ports with switches with N ports you need to take 4 switches in the front with N/2 ports and 2 switches in the back interconnecting all the ports to the back for a total of 6 switches, so not even including cables a modular switch providing 2N ports is worth 6x per port than a fixed switch... The number are a little better for bigger but approach at a cost per port of 3x per port not including cables. In my experience modular switches are more in the ball park of 2.5-3x per port. I am not including other possible benefits that modular switches can have non-blocking fabrics.
The real debate is between OSPF and QSFP-DD as they are the only ones with a shot at high density.
QSFP-DD is backwards compatible so that you can run your 400G port as a 100G or 40G with 4 lanes at 25G/10G. OSFP will be able to do the same thing but with an adapter from OSFP to QSFP
But what this article fails to mention is that OSFP is the connector of the future that will support 8 lanes of 100G to support 800G ethernet with good signal integrity and capability to support high power optics for longer distances. ( This looks to be in 2020 )
So QSFP-DD is a 1 generation connector...
OSFP will see us through 400G and 800G.
Disclaimer I work at Arista Networks who is bullish on OSFP.
For more info see Andy Bechtolsheim's talk on 400G optics at OCP
I am also optimistic about OSFP as compared to the QSFP size for 400G. I'm not a laser engineer but I do have a pretty good understanding of how 400G will be achieved through two strands of singlemode. For medium reach optics it's actually an 8-channel CWDM with prism built into each optic, so the optic body needs to be large enough to dissipate the heat from eight separate 25 Gbps lasers. CFP size will obviously handle that but is a true first generation solution and is huge.
The only bad thing about OSFP is that its highly similar name will confuse the hell out of network engineers who never see the OSI layer 1 of a network, and deal with things at OSI layer 3.
for long reach, yes, but not for ISP-to-ISP interconnection... the cost of coherent (QPSK, 8PSK, 16QAM) 100G SerDes and modulation/demodulation is considerably higher and only justified if going a great many km. For intra-peering-facility connections I don't see coherent 100G (or multiples of coherent 100G bundled by CWDM into 200G, 400G interfaces) becoming the norm anytime soon.
I was looking at this the other day; it sounds like 100G-DR1 and 400G-DR4 are not coherent and vendors are aiming to make it commodity. Coherent is more like 200-600G per lambda.
At present 100G-LR4 is the defacto standard for ISP peering interconnections (such as to an IX switch in the same building, or for PNIs between big players within the same carrier hotel). Because it is very low cost. The cheap 25 Gbps x 4 approach and low cost cwdm prism are a big part of that. Since 100G will soon move from qsfp format down to the same size as SFP and SFP+ (fitting 48 in a 1RU height, 17.5" wide line card in a chassis based Arista), I don't foresee them becoming any less popular... Not when the optics are so affordable.
We aren’t talking about long distance and coherent here but direct detect. Coherent is not cost or power competitive with direct detect for short reaches yet.
When I talk about SERDES I’m talking about the electrical signal interface to the optic. Also it’s worth noting that the majority of coherent DWDM interfaces are some kind of embedded optic already.
There is no real debate between QSFP-DD and OSFP. OSFP will exist at Google but nearly everyone else is going to use QSFP-DD. The main reason is backwards compatibility for data center upgrades. The only reason to use OSFP might be because you want to use a 400G-ZR plug. But with the advent of 7nm DSP, vendors will be able to fit the ZR into the DD plug.
So if the above is true OSFP is not needed now.
Honestly, 800G is only incremental at best and frankly it’s not clear that An 800G pluggable is the way to go. The 800G presumes 32 radix and that breaks down power-wise in the datacenter when we get to spine switches where we need 128 radix. The reason being that we need so many power hungry chips to make a high radix switch fully non-blocking (eg Clos topology). 112G SERDES will have limited reach and hence require many retimers which eat power too. So the OSFP is in my view is A solution looking for a problem.
I get the attraction of OSFP, but in early days backward compatibility usually trumps future potential. If I'm buying an high density switch/router tomorrow (figuratively speaking), I'd greatly prefer the flexibility to use the density it provides for some existing lower speed connections. As long as QSFP-DD 'works', as a customer I'm not really concerned with which technology is more elegant or technically 'better'.
The problem this presents for OSFP going forward is that it encourages early adoption to steer toward QSFP-DD, which will tend to drive follow-on purchase decisions as well.
Sorry but OSFP has at most a life of 2 generations and likely only useful for a single generation. 112G SERDES is not going to scale well and will require repeaters all over the place. Moreover doubling the signaling rate to 224 Gb/s is wholly impractical. The future is embedded devices and not legacy pluggables.
Hopefully you guys have ECC on your 64MB of SRAM, otherwise the meant time to bit flip due to Single Event Upset (SEU) is around 400 days ( based on 200 Fit/Mb/Billion Hours from previous experience ).
No ECC on chip, but we do have column redundancy. We are pushing the envelope in terms of SEUs, making an assumption that the right programming model and run time will be able to compensate for high soft error rates. It's a contentious point, but basically our thesis is that with 1024 cores on a single chip, cores are "free" and it "should" be possible to avoid putting down very expensive ECC circuits on every memory bank (x4096). Some of our customers don't notice all bit flips because they have things like Turbo/Viterbi ..channels aren't perfect...
Some datacenters enable Priority Flow Control (PFC) which is different in that it pauses only the traffic with a specific PCP ( Priority in 802.1Q vlan tag ). They assign storage traffic a specific vlan priority and treat it as lossless with flow control but the rest of the traffic is unaffected.
The mechanism here Pause is an abomination which should never be enabled.
> Some datacenters enable Priority Flow Control (PFC) which is different in that it pauses only the traffic with a specific PCP ( Priority in 802.1Q vlan tag ). They assign storage traffic a specific vlan priority and treat it as lossless with flow control but the rest of the traffic is unaffected.
I don't think he had his TV in a datacenter.
From the article:
> After some clever deductive reasoning, a.k.a randomly unplugging cables from the router, I determined that my TV was sending these mystery frames (yes, my TV — I have a Sony X805D Android TV).
> The mechanism here Pause is an abomination which should never be enabled.
What? You can't be serious, I think you have no idea what that would cause in almost every ethernet network. Let me tell you: a lot of packet loss that messes with TCP streams etc.
L2 pause frames are used by practically all of the ethernet devices, and for a really good reason. Pause frames are a perfectly good way to do flow control in most networks. Not having them means a lot of lost frames and generic pain in most networks... except of course datacenters.
Sure, it's not a standard. But it's good enough for 99.9% of use cases. Just maybe not in datacenter.
Ah... The problems of crappy consumer ethernet equipment ( I work at an ethernet switch vendor so excuse the rant )
What is likely happening is that your switch is configured by default to implement both rx and tx pause. What is happening is that your TV who's also erroneously ( in my opinion ) configured to transmit pause goes bonkers, starts sending pause to your switch. Your switch then starts buffering packets for your tv until the buffers are full and then starts transmitting pause to everyone else including ports. The switch must have some horrible buffering policy where one port ( the tv port ) can hog all the buffers and deprive every other ports of being able to send...
Now the kicker is that the way every endstation implements pause is this... Notice the pause quanta in the Pause packet is in units of 512 bits and in the packet you captured it is set to the maximal value of 65535 which is on a 100Mbps port ( presuming 100Mbps since the Mediatek has 4x100Mbps (Fast Ethernet) and 2x1Gbps ( Gigabit Ethernet ) that computes to >>> 512*65535/100.e6 = 0.3355392seconds
A normal Pause sends this packet periodically and once it has buffers to receive will send a pause with a quanta of 0 meaning cancel previous timer... but if it's malfunctioning who knows if it ever will...
The sad part is that I don't even know what to recommend for a good consumer level switch that has good defaults or configurable defaults and sane buffer config... Mine is a dinky one probably vulnerable to this problem as well... Need to do some research.
> A normal Pause sends this packet periodically and once it has buffers to receive will send a pause with a quanta of 0 meaning cancel previous timer... but if it's malfunctioning who knows if it ever will...
As with most protocols, it doesn't work if it's not implemented properly.
Many hardware implementations have no knowledge of when the buffer can be emptied, so it's understandable they treat it as a on/off switch. Screaming, my buffer is full, don't send me anything for 65535*512 bit-times! Which is perfectly good, because otherwise all those incoming frames would need to be dropped anyways.
Remember, small embedded devices can't often guarantee ethernet DMA slot time to DRAM, and definitely can't afford to have a dedicated DRAM channel, so those buffers are on a 2-8 kB on-chip SRAM block or equivalent.
When the buffer is "full" (above high water mark), an interrupt gets generated and device firmware will set up an appropriate DMA transfer to empty it. Once that is done, the device should of course send a PAUSE 0, and all is good.
> The sad part is that I don't even know what to recommend for a good consumer level switch that has good defaults or configurable defaults and sane buffer config...
Like you must know, you can turn it off entirely in most managed switches, see what happens to data transfer speed.
Most consumer level gigabit switches seem to have maybe 16 kB buffer. So they don't really have much buffers (or anything) to configure.
Like you said the host might have small buffers and without Pause it would drop, but who's supposed to buffer the packets, the cheap switch with 16kB of buffers and super idiotic buffer configuration such that everyone else on that switch gets paused?
You seem to think that it's bad to drop packets in the nic and while some nic might have buffers that are too small but in general you should drop. If you use TCP the window will adjust to whatever your bad nic and embedded system can handle. At least you won't affect the others by spreading pause like a cancer ( can you tell I am cynical on pause )
Usually on a switch you can usually drop packets based on the number of packets destined to a port and packets buffered per input port. This is how you can avoid head of line blocking but again if you are right with 16kB that's barely enough for a jumbo packet (~9200B)... geez that's depressing.
> If you use TCP the window will adjust to whatever your bad nic and embedded system can handle.
TCP window, sigh... It can't deal with the situation where, say, every second frame is lost, because someone thought 2 kB is enough buffer. TCP congestion control mechanisms are great for actual congestion, but when packet loss is due to other causes, it's actually pretty bad.
Again, TCP is no substitute for flow control in this case.
Doesn't matter how nice NIC you have. The problem usually happens before the packets reach your nice NIC.
For that reason you should basically never enable ethernet flow control except for on a Fibre Channel over Ethernet SAN, and even then they had to invent Priority-based Flow Control to make it sane. If this is a managed switch then you should be able to disable it.
(I used to also work at an ethernet switch vendor.)
I've had the exact same thing happen when a host crashed. Relatively modern Intel network chip on the host, Netgear GS108 switch (BCM5398 I believe). Presumably when the host stops servicing interrupts, the card's buffer fills up and then generates pause frames.
I don't think it requires the switch to have a bad buffer policy - all the switch ports didn't die at once, just one-by-one as each connected device tried to send a broadcast packet. I don't see a way of avoiding this logical situation if pause frames are sacrosanct - it seems that a switch would need a heuristic to forget pausing and start silently dropping packets to only the affected ports.
(I've since disabled pause frames on those cards, since I don't really need them.)
This same root issue - trying to implement "reliable multicast" - is why DaveM rejecting the AF_BUS IPC implementation a few years ago. In any multicast or broadcast system you can't allow one stoned endpoint to wedge the bus for everyone.
> The switch must have some horrible buffering policy where one port ( the tv port ) can hog all the buffers and deprive every other ports of being able to send...
Or could it be that the switch is oblivious to STP ethernet addresses and PAUSE frames ? The frames shown(presumably originating from the TV) have a destination address of 01:80:C2:00:00:00 , and if the dumb switch doesn't know that this is kind of a special address, it'll just do what multicast bit in that address tells i to do - copy the frame out to every port..
Yeah. IMO most of the blame here falls on the TV rather than the switch. Even when it's implemented well, Ethernet pause frame generation should not be enabled - certainly, not by default - on a consumer product, because it's really unreasonable to expect the average $17 consumer switch to handle it nicely. Furthermore, there's usually little need & little benefit to trying to make a home network lossless via L2 pause.
I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible.
The fact that it is a a specific interval indicates a stuck bit in memory.
Some good previous public stories about such incidents https://www.verizondigitalmedia.com/blog/being-good-stewards... https://twitter.com/cperciva/status/1309568337408454658