
What's the max speed on Ethernet? - jessaustin
http://blog.erratasec.com/2013/10/whats-max-speed-on-ethernet.html
======
mcpherrinm
Linux isn't the only limitation on line rate traffic. Many gigabit Ethernet
switches can't handle line rate min sized gigabit traffic. Mine is an
otherwise good Asus switch that only does 1,000,000pps instead of 1,488,095
pps. I would bet similar restrictions could exist in Ethernet cards too.

~~~
wmf
Yep, I read that the Intel 82599 can/can't (I forget which) hit line rate for
packets that are a multiple of 64 bytes. (Care to guess the DMA burst size?)
These kind of quirks have been documented in almost every NIC.

------
jws
He calculates 476mbps at the IP level on 1gbps ethernet, but continues…

 _Note that this disparity only happens because we are using very small
packets, where per-packet overhead dominates. In normal Internet usage,
packets are 500-bytes in size, and the per-packet overhead is usually
ignored._

So, to one digit of precision, gigabit ethernet is still gigabit ethernet.

~~~
honeypoopoo
Packets are not 500 bytes in size, not even as an average (mean) or median.
Many studies of frame length distribution show 64 bytes (for the Ethernet
frame) as the most common length, probably because of TCP ACKs. Next most
popular is 1518 bytes, the Ethernet maximum, probably because of bulk
transfers over TCP.

Also, jumbo frames are technically not Ethernet. The IEEE revised the 802.3
spec to increase the maximum envelope from 1518 bytes up to around 2000 bytes
to accommodate VLAN stacking and other encapsulating schemes. But jumbos --
usually 9000 or 9216 bytes -- aren't legally Ethernet.

~~~
mino
Correct: the packet size distribution over any aggregated Internet link is
typically bi-modal: a peak at around 64 bytes (acks) and one at around 1500
bytes (transfers that fill the MTU).

The article seems to me very approximate.

------
ddoolin
Anyone familiar with why Linux in particular can't hit the limit?

~~~
recuter
I believe it has to do with the penalty going between kernel and userspace
because I remember repeatedly seeing novel projects that blow past these
limits suggest pushing the whole networking stack to userspace. But I could be
wrong.

Interesting question, hope somebody chimes in.

~~~
ajross
Could very well be hardware- or driver-dependent, and obviously the switch or
device on the other side of the cable has a say in it too. There's an awful
lot of pluggable stuff in there to support everything in the market. Really
this thing needs to say "I only measured 1.3Mpps between two 3.11 kernels on
IVB boxes with RTL8168 devices connected via a Netlink blah blah blah switch"
to have any diagnostic meaning.

------
xtacy
The limitation you see on the number of pps is mostly due to software and
hardware artifacts. With careful tuning (interrupt affinities, etc.), or
bypassing a bunch of software layering (e.g. Netmap), or special stacks (e.g.
Intel DPDK) you should be able to hit "line rate" packet processing.

------
2bluesc
No mention of MTU?

This is why there are jumbo frames for bandwidth (not latency) intense
operations.

~~~
sliverstorm
I'm told modern gigabit ethernet devices derive practically no benefit from
jumbo frames, and the pitfalls of using them are legion.

~~~
liotier
Each frame is a CPU-hogging interrupt so if your bottleneck is the CPU, then
jumbo frames will help by moving more data for the same number of frames. This
happens easily on routers with low-power CPU.

~~~
fulafel
This is only true for very basic Ethernet hardware. Various interrupt
mitigation schemes came along in the late 90s along with gigabit ethernet.
Even low-end Realtek gigabit chips have various segmentation offload and
interrupt coalescing features these days.

~~~
acdha
Those schemes have varying levels of effectiveness. I personally experienced
this wasn't a given even as late as the 2005-2010 era, when our purchases were
shifted towards certain models specifically due to the issues with drivers or
buffer limits.

I learned never to underestimate how much functionality a major vendor will be
willing to cut if it saves $.10 per unit shipped…

------
cstrat
Nice write up, so you're saying that even though you can push the limits of
Ethernet locally, it won't help you once you hit a router.

I would love to have access to a Gbps connection to the internet... stuck with
my 15Mbps ADSL.

~~~
NamTaf
15Mbps? LUXURY.

Nice write-up. I wonder if there's any market in spec-breaking routers that
can accommodate smaller inter-frame gaps, etc.? What use is there for this
that is not covered by jumbo frames?

~~~
hobs
Do you mean for internal use only? Wouldn't that be a moot point once it hits
the next hop?

~~~
toast0
There's not necessarily one next hop. If you can stuff more packets than
standard out to your upstream, and the upstream delivers the packets over
multiple interfaces, it can pad the packets out to the standard length, and
you get > 100% utilization.

I understand the original purpose of the minimum packet length (needed for
reliable collision detection), but I don't think it's really necessary for
gigE, where I believe full duplex, point to point connections are mandated. I
also don't think the minimum size would be sufficient to guarantee collision
detection in a hypothetical shared medium at the allowed bitrate and
distances. Frame to frame delay on the other hand, seems like it would still
be useful , although maybe it could be tuned.

~~~
hobs
Good answer +1, I was sort of thinking about this but I didnt have the
knowledge.

------
EB5
The speed of light. (299,792,458 metres per second)

