
Benchmark results of Kubernetes network plugins over 10Gb/s network - homarp
https://itnext.io/benchmark-results-of-kubernetes-network-plugins-cni-over-10gbit-s-network-36475925a560
======
tgraf
[Disclaimer: I'm one of the Cilium authors]

We have been trying to reproduce the performance results ever since the
article was published as this is not in line at all with what we measure daily
in our CI. We can easily do a multiple of these numbers.

There are some obvious flaws in the benchmarking scripts [0] such as using the
"used" column of `free` without taking into account cached file buffers.

However, it does not explain why HTTP and FTP are worse compared to the TCP
benchmark which is doing wire speed at ease. None of the Cilium datapath is
HTTP or FTP specific unless HTTP specific security policies are in place in
which case HTTP traffic is actually parsed.

We have requested more information on the scripts used by the author and
continue to investigate. We will publish results as soon as we can reproduce
this.

As stated by other commenters as well, most of these benchmarks are measuring
the same Linux kernel code except for Weave (OVS) and Cilium (BPF). However,
at the specified MTU of 9000, the bottleneck for all plugins will not be the
forwarding datapath but the actual client and server code copying the data in
and out of the kernel as there are very few packets actually being created and
forwarded.

[0]
[https://gist.github.com/AlexisDucastel/ebb884831aeec5827e4df...](https://gist.github.com/AlexisDucastel/ebb884831aeec5827e4df82975e5c4eb)

~~~
nicolast
> most of these benchmarks are measuring the same Linux kernel code

This, 1000x this. I'm afraid too many people treat their CNI plugin as 'magic'
whilst many of them really aren't. 'Host' versus Calico is basically
benchmarking the impact of a Linux bridge device, and maybe some more iptables
rules than the host has (depending on whether the host benchmark has iptables
enabled at all, whether there are K8s network security policies in place and
enforced by Calico,...).

Also, configuration details are lacking. E.g. in the Calico benchmarks, was
ipip enabled or not?

~~~
krakrnews
Yes most solutions are using Linux kernel, so what's being measured is indeed
the impact of the way the kernel is being configured to achieve container
networking. But that doesn't make those design choices, and the tests,
meaningful. Calico, for example, contrary to your assumption, uses neither a
Linux bridge device nor iptables for packet forwarding. (It does use iptables
for policy enforcement, but that's not being tested here.)

~~~
nicolast
I'm aware it doesn't use iptables, except to implement network policies, hence
the reference. Good call about the bridge usage, my bad, makes sense that's
not being used given Calico is L3...

------
zaphar
I wonder what the latency stats are for each. Bandwidth per second is useful
but in my experience latency is the more important statistic for the
applications I manage.

~~~
throwaway2048
Packets per second aswell, speed testers generally give you the absolute best
case full MTU sized packets for 100% of the duration.

Real world traffic has lots of nasty small packets.

------
scurvy
Any plans to run some packets per second benchmarks with latency? Everything
today can run at line rate, especially with jumbo frames. These existing
benchmarks don't show much. Networking people want to see how many PPS you can
push, at what latency, and at what packet sizes can you achieve line rate?

~~~
kingosticks
Some IMIX benchmarks would be a good start.
[https://en.wikipedia.org/wiki/Internet_Mix](https://en.wikipedia.org/wiki/Internet_Mix)

------
adamtulinius
I find it slightly weird to test different protocols like that, especially the
SCP test, where the author states:

"With SCP, we transfer the 10GB random file over scp using OpenSSH server and
client. We can clearly see that even bare-metal performance are much lower
than previously."

SCP is just notoriously slow, and it should be expected to not be able to max
out the connection with it.

~~~
chrismeller
No one is saying you should use SCP for anything or that it's an alternative
to any of the others - none of the tested protocols are practical alternatives
to each other. It was included to provide a comparison point for the encrypted
version of WeaveNet.

Encrypted WeaveNet handled plain TCP traffic at 1,330 Mbit/s. Without the SCP
comparison there's no way to tell whether that's really really good or really
really bad. Seeing that the unencrypted version of WeaveNet handled SCP at
1,594 Mbit/s shows that it's not horrible, but could probably be better. It's
good for WeaveNet folks to know they have some room for improvement and it's
good for you because you might want to consider alternatives if you require
encrypted traffic and are very bandwidth sensitive.

~~~
bboreham
[Note I work on Weave Net]

When we ran a similar test we got 3.75Gbit/sec; the figure in this article is
closer to what we got in our "slow mode" which is used as a fall-back when it
can't get the in-kernel path to work.

[https://www.weave.works/blog/weave-net-performance-fast-
data...](https://www.weave.works/blog/weave-net-performance-fast-datapath/)

------
stevenacreman
For those interested in a feature comparison I did one here:

[https://kubedex.com/kubernetes-network-
plugins/](https://kubedex.com/kubernetes-network-plugins/)

~~~
gtaylor
This is pretty neat. Would be interested to see aws-cni-vpc in there, too!

~~~
stevenacreman
Added as the final column in this sheet.

[https://docs.google.com/spreadsheets/d/1qCOlor16Wp5mHd6MQxB5...](https://docs.google.com/spreadsheets/d/1qCOlor16Wp5mHd6MQxB5gUEQILnijyDLIExEpqmee2k/edit?usp=sharing)

------
nicolast
As mentioned here, the article doesn't specify whether the Calico setup uses
ipip or not which could have a measurable impact.

In the Calico-without-ipip case, you're basically comparing host networking
(let's assume with some iptables enabled) with host+iptables+bridge networking
(yes, there may be more iptables rules involved). Let's assume the impact of
iptables is the same between both, then it'd be interesting to measure the
impact of the Linux bridge being used. As a colleague of mine mentioned, this
may be barely noticeable on a 10Gb interface, but could be on faster networks.
How about running these tests on a 20Gb or 40Gb network? These are quite
common in datacenter networks, which is where you'll be using these CNIs
(unlike GKE, EKS,... where you can integrate with the 'native' SDN).

Finally, I'd be interested to get some results when using the macvlan CNI
plugin (though then one loses network policy support, sadly enough :( )

~~~
barbecue_sauce
I don't work in the networking space, is ipip synonomous with IP-in-IP
tunneling? (I can't find a definitive answer in my cursory attempt).

~~~
X-Istence
Yes.

[https://docs.projectcalico.org/v3.2/usage/configuration/ip-i...](https://docs.projectcalico.org/v3.2/usage/configuration/ip-
in-ip)

This is used when there is source/dest filtering happening on the network
(such as in certain cloud providers where you can't send packets from
10.0.0.1/32 to 192.168.1.2/32 because 10.0.0.1/32 is not valid, so you pack it
into another IP packet that makes the source seem like 192.168.1.3/32).

------
gweeed
Hm, running iperf3 stream is pretty synthetic benchmark, what about real-life
workloads? iptables should be pretty damn slow the more rules are added as
they are walked linearly, I doubt this has been properly evaluated in the blog
post above.

------
shaklee3
For those looking to go above 10Gbps, Intel makes a dpdk plug-in, and Cisco
makes the VPP virtual switch with dpdk as a back-end.

------
hacknat
Flannel has encryption. It specifically has a documented IPSec mode. I’ve
actually run it in production.

~~~
sascha_sl
it also has a very very simple wireguard plugin that "just works" in my
experience. You can read the entire thing in a minute.

[https://github.com/coreos/flannel/blob/master/dist/extension...](https://github.com/coreos/flannel/blob/master/dist/extension-
wireguard)

------
throw2016
Things like CNI take away from the development and understanding of general
networking technology to scripts and json/yaml wrappers that not only add
another layer of complexity but also prevent users from understanding what
they are actually using.

For instance in this post itself we learn about all these CNI wrappers but
learn nothing about Vxlan, BGP and Linux networking tools that these are using
underneath. And those are actually easy to use directly. This does a
disservice to both developers and users.

For instance true game changers in this space is tech like Wireguard that can
deliver encryption at near line speed and would kill these benchmarks.
Everyone can benefit from Wireguard today. Initiatives like CNI encourage
wrappers and those then wrapping tools like Wireguard end up getting the
recognition and success while the underlying tools like Wireguard fade into
the background. This does not seem like a healthy or sustainable development
model.

~~~
wmf
K8s is all about encapsulating devops practices into code, so it's kind of
philosophically totally opposite to learning all the details of virtual
networking and configuring it from scratch.

If Wireguard doesn't write the wrapper themselves they have no one to blame.
People need to understand that the hard/cool part and the valuable part aren't
the same and if you are too cool to create value then you can't complain about
other people taking it.

~~~
throw2016
This is a nice idea about abstraction but without knowledge you cannot run
anything professionally with any degree of reliability or confidence. What
happens when things break? Google, Facebook and any company dependent on
delivery have full fledged networking and ops teams.

All these wrappers still require time investment in knowledge, setup and
configuration. The cost for users knowing the wrappers but not the tech is
being locked in a platform, if they know the tech even if Kubernetes crashes
and burns their knowledge and time investment remains valuable.

Before Wireguard it was extremely difficult to get near line speed with
encryption, now accessible to all. If this is not 'value' what is? Given this
post is about networking benchmarks this is a huge deal. If people stop
creating these kind of networking tools and technologies then there is going
to be nothing to wrap.

------
polyomino
In many cases these containers are running on the same physical machine. Could
the network stack be optimized so much so that writes from one program inside
a container can write directly into pre-allocated memory of another container
using our existing networking interfaces?

~~~
tgraf
[https://www.slideshare.net/ThomasGraf5/accelerating-envoy-
an...](https://www.slideshare.net/ThomasGraf5/accelerating-envoy-and-istio-
with-cilium-and-the-linux-kernel)

Slide 21-25

------
kklimonda
Was calico ipip encapsulation enabled or disabled?

------
sofaofthedamned
I'm impressed that all non-encrypted methods are so close to bare-metal.
Wonder how much CPU was impacted at those rates though?

~~~
kitd
CPU usage was the final test, near the end of the article.

~~~
sofaofthedamned
Ah missed that, ta!

------
jamp897
I have to admin I’ve never quite understood the need for the overlay network,
esp with such a large performance impact. Security has been moving higher up
into the app layer anyway with Beyond Corp style approaches. What are the uses
cases driving it?

~~~
dharmab
In our case, service mesh and network isolation (including for potentially
untrusted code). Being able to ensure that pods can't talk to each other by
default and managing whitelists for service to service communication. Managing
that without an overlay is a pain once you're past a handful of nodes.

------
scurvy
Also, is Romana still actively developed? It seems like they had a lot of
momentum, then suddenly things went radio silent on their Slack and git repo.

------
godelmachine
Has not
[https://news.ycombinator.com/item?id=18564340](https://news.ycombinator.com/item?id=18564340)
achieved the same result?

------
ravedave5
I wish there was a similar comparison of proxies.

------
londons_explore
All these 'plugins' are simply scripts and tools which configure the linux
kernel iptables and network forwarding rules right?

None of them actually handle packets, connections, routing etc directly I
assume?

Given that, it seems disingenuous to benchmark them... You're simply
benchmarking the Linux Kernel vs the Linux Kernel.

~~~
whalesalad
At the end of the day you’re benchmarking a strategy. It doesn’t matter that
Linux is the underlying thing executing that strategy. All of these tools have
branding and marketing around them... It’s important to see which ones live up
to claims.

