
FreeBSD has lower latency, and Linux has faster application speeds - krn
https://www.quora.com/Is-FreeBSD-faster-than-Linux/answer/Ben-Lunsford?share=1
======
drewg123
Speaking as somebody who works on the FreeBSD kernel at Netflix, the whole
article is kind of nonsensical. We don't use FreeBSD because of "latency". We
serve clients over https that could be tens or hundreds of ms away using the
kernel TCP stack. The folks who care about latency are high frequency traders,
who care about every nanosecond. They tend to use userspace / hardware
offloaded solutions and do things like busy-waiting for messages.

~~~
tiffanyh
Drew

Have you ever consider using Dragonfly. It appears to dramatically beat in
perf test FreeBSD (2017).

[https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf](https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf)

I’m curious to hear your take on Dragonfly since you’re a kernel developer.

Thanks in advance for all your work.

Edit: more recent links (2018) with even higher perf

[https://leaf.dragonflybsd.org/~sephe/dfly1.pdf](https://leaf.dragonflybsd.org/~sephe/dfly1.pdf)

[https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf](https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf)

~~~
drewg123
I have looked at Dragonfly with interest, due to how they manage VM. I really
like how they shard the page queues to be per-cpu. With that said, a lot of
work has been done by jeffr in the last 6 months to a year to improve NUMA
performance (which we partially sponsored) which has dramatically improved VM
scalability even on single socket boxes, and which has allowed us to remove a
lot of our local hacks.

I considered giving Dragonfly a try in the past, but just never had the time.
There would be a depressing amount of work before we could even consider
Dragonfly, mostly centered around our async sendfile (which is now upstream in
FreeBSD), our TCP changes: BBR (not yet upstream), RACK (now upstream) and TCP
pacing (now upstream). Not to mention unmapped mbufs (not yet upstream) and
kernel TLS (also not yet upstream). Also, the last time I looked, Dragonfly
did not have drivers for Chelsio or Mellanox 10/25/40/100GbE NICs. Without
even just one of these things, performance will be so bad that any comparison
testing would be meaningless.

~~~
tiffanyh
Thanks for the thoughtful reply.

Just a thought, the Linux/BSD community might hugely benefit if someone with
your deep knowledge released a set of perf test scripts that anyone can run
locally to regression test network perf. That way, the OSS community can
integrate those perf test scripts into their commit/regression test pipeline.

~~~
toast0
It would be great to have this kind of test, but often the regression test is
going to be run the system on production load. Benchmarks have a way of
testing something, but not quite what you need to test. The interactions
between the whole system are important.

For the Netflix boxes, they're pushing 40gbps+, not a lot of the community is
going to be to be able to test that, unless they have fairly expensive
networks laying around.

~~~
drewg123
You're spot on. We don't really have good benchmarks that approximate our
production traffic.

BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :)

~~~
jacquesm
As someone who grew up around Telex machines and current loop interfaces:
consider me impressed!

------
rb2k_
No sources or benchmarks?

Also:

> But unless you’re using Fedora, your servers aren’t gonna taste it soon.
> Just by bad timing, the new Ubuntu LTS (18.04) due out next week will still
> use 4.15, which it will support for 5 years. Since CentOS and Debian are
> even further behind on kernels, you won’t see a lot of “fast Linux” in
> production until April 2020 when Ubuntu 20.04 LTS comes out.

If you are at a point where the differences would matter to you, you can (and
will) probably just install a newer kernel and have the improvements available
today. Just because upstream doesn't have it doesn't mean you couldn't compile
it yourself or use backports.

~~~
ams6110
> Just because upstream doesn't have it doesn't mean you couldn't compile it
> yourself

True, but if you are managing production machines, compiling kernels for every
errata and security update is pretty low on the list of things you get
enthusiastic about. Also a lot of commercial or scientific linux software is
only "certified" and supported on stock RHEL kernels.

~~~
geofft
On the other hand, if you're making or losing money on latency (e.g., you're a
high-frequency trader), compiling kernels for every potential performance
improvement _is_ a thing you're enthusiastic about, a lot of security fixes
(local privilege escalations, drivers for desktop hardware, etc.) don't apply
to you, and you don't care about things being "certified" because you have the
expertise to fix problems in-house as necessary.

And from the other direction, if you're making or losing money on security,
expecting distro kernels to turn around security fixes in a timely manner
seems like a mistake. At least know _how_ to rebuild your distro kernel with a
local patch.

------
sliken
The post seems to be confusing FreeBSD userspace (which OSX adopted parts of)
and the FreeBSD kernel (which has the mentioned lower network latency). OSX
didn't adopt the FreeBSD kernel, but instead started with the Mach microkernel
and has been going in their own direction since.

~~~
4ad
Not that it matters (the post is mostly just unfounded opinion), but:

> OSX didn't adopt the FreeBSD kernel, but instead started with the Mach
> microkernel and has been going in their own direction since.

MacOS kernel uses a combination of Mach and BSD kernel code. It started with
4.4BSD, and at some point the bulk of BSD code was updated with code from
FreeBSD 5.

After that, it was piecewise updated with more modern stuff, the MAC framework
(used to implement sandboxing on iOS and macOS) and DTrace from newer FreeBSD
kernels.

~~~
hestefisk
There is a great talk on OS X internals from CCC here:

[https://events.ccc.de/congress/2007/Fahrplan/attachments/105...](https://events.ccc.de/congress/2007/Fahrplan/attachments/1053_inside-
macosx-kernel.pdf)

It particularly talks about the unique hybrid kernel approach and how there is
nothing like it.

~~~
loeg
BSD itself took the VM subsystem from Mach (so BSD is kind of an interesting
hybrid as well). While it has been heavily modified in the last 25+ years, the
VM portion of the FreeBSD kernel is still stylistically distinct from BSD
portions of the kernel source code.

------
TorKlingberg
With such a long post, I would have expected some technical details for _why_
FreeBSD has lower latency, and Linux has faster application speeds.

------
vermaden
FreeBSD was used in 1999 to render The Matrix on 32 Pentium II boxes because
the software in Linux Compatibility mode on FreeBSD was faster then natively
on Linux, that is a fact:

[https://www.freebsd.org/news/press-
rel-1.html](https://www.freebsd.org/news/press-rel-1.html)

FreeBSD can be several times faster then Linux when it comes to network stack:

[https://pbs.twimg.com/media/CzFfTSRUQAATwaq.jpg](https://pbs.twimg.com/media/CzFfTSRUQAATwaq.jpg)

But often Linux is faster, you just need to find benchmark that favorites one
or another, both are fast in general.

~~~
ajross
Why on earth would you link to a Phoronix screenshot instead of the article
that explains it?! Those results look suspicious enough[1] that I wanted to
look it up. But of course I can't, and Google isn't helping me much.

Shame! Shame, shame, shame!

[1] Factor of 3+ differences not just between BSD and Linux but between
otherwise very comparable linux distros. Something's weird with the setup
there, like it's measuring default firewalling overhead or something and not
kernel behavior.

~~~
buzer
Looks like source is
[https://www.phoronix.com/scan.php?page=article&item=netperf-...](https://www.phoronix.com/scan.php?page=article&item=netperf-
bsd-linux&num=1)

I guess they have everything just set to default disto settings which likely
in some cases include firewall while others don't. They also don't mention
which NIC they are using, the motherboard has two different NICs (i218-LM &
I210-AT).

~~~
jasondclinton
Those benchmarks were run on machines with different processor clocks and
different motherboard chipsets. The results are invalid.

------
api
A bit of a tangent but: take another look at FreeBSD folks.

If we re-did our infrastructure from scratch we'd look at either FreeBSD or
Alpine Linux. Both would offer an escape hatch from the needless bloat that
mainstream Linux has become.

~~~
hpcjoe
Dependency radii of packages in CentOS are horrible. Your minimal installs are
hovering above 1GB. Debian are better at 400MB or so. Alpine is great, in that
you can get this down to well under 100MB.

This has been a long standing complaint of mine[1], and an argument I've had
with many proponents of "software must come packaged only from the provider of
the distro" people.

Physical and logical footprint is a real thing. Ridiculous secondary/tertiary
dependencies are a waste of that precious resource, and an unneeded/unwanted
installation of useless bits.

[1] [http://scalability.org/2018/04/distribution-package-
dependen...](http://scalability.org/2018/04/distribution-package-dependency-
radii-or-why-distros-may-be-doomed/)

~~~
mmt
Although I agree with you that this bloat is undesirable, I'm not convinced
it's actually a problem with the distro. (My primary experience is with
Ubuntu, and to a lesser extent, mostly by extension, Debian)

Rather, I think it's a problem with the package maintainers themselves. I
certainly admit that for very many packages, this is a distinction without a
difference.

However, my point is that this dependency bloat isn't based on some kind of
policy that the distro is encouraging or even enforcing. Rather, the problem
is merely that too many source packages create too few targets, perhaps
because the original packagers never foresaw the need (for, e.g., both an X
version of some graphics or GPU-related library and a non-X one for pure
server work).

I've occasionally gotten around this problem by custom-building a local
version of the package that avoids the dependency bloat. That violates the
"software must come packaged only from the provider of the distro", unless one
considers the local superset a forked distro.

> Physical and logical footprint is a real thing.

Ultimately, I think you and I are in a vanishingly small minority here.
Virtualization (even the full, pre-container kind, with a full copy of the
kernel on every instance) gained a startling level of a popularity, and was
even lionized as a money-saving tool (which, of course, it was, for some "IT"
shops).

------
parvenu74
> "There are some 'BSD-inspired' networking improvements to Linux starting
> with the 4.16."

By "inspired by BSD" do the author mean the code has been copied and pasted
from FreeBSD to Linux, something which cannot be done in reverse because of
the encumbrance of the GPL?

~~~
zeth___
BSD is free to adopt the GPL at their leisure. Who knows, companies might then
start contributing to the project instead of building OS's on top of it,
looking at you osx.

~~~
parvenu74
GPL supporters are all about freedom -- until someone advocates for the
freedom to make a proprietary product on top of the open source base.

~~~
bonzini
More specifically it's about freedom for _all_ users, transitively.

~~~
ComputerGuru
Except for developers who wish to use GPL code without it virally requiring
them to relicense everything under GPL, that freedom does not exist.

~~~
bonzini
The key word is transitively. Those developers can do everything except
depriving _their_ users of freedoms originally granted by the copyright
holder.

------
j16sdiz
The benchmark use outdated API.

Misleading on distro release a date.

No explanation on the improvements.

I would vote down this post if I could.

~~~
josersanchez
You can down vote on quora.

------
snvzz
Doesn't Dragonfly (freebsd fork as of 2003) have even lower latency?

------
ksec
I reckon if someone make Java a first class citizen on FreeBSD, it may
actually get a larger amount of usage.

------
HippoBaro
Woudn't anyone really into IP latency optimization be using something like
DPDK on Linux? If you want high-performance networking, then using the kernel
isn't the best idea IMO.

------
stefan_
Latency and throughput ("application speeds"), certainly in computing, are
competing qualities.

------
rjmcmahon
Well, here's just one measurement using iperf 2.0.12. All clocks are
synchronized to a GPS disciplined oven controlled oscillator (OCXO). Connected
via 1Gbs. FreeBSD latency is significantly better.

Source fedora 28 xeon machine, receivers FreeBSD 11.1 and Fedora 25 (same brix
platform) connected via a Cisco 300 switch.

RX:

FreeBSD: root@zeus:/usr/local/src/iperf2-code # iperf -s -u -e --udp-
histogram=10u,100000 --realtime
\------------------------------------------------------------ Server listening
on UDP port 5001 with pid 53703 Receiving 1470 byte datagrams UDP buffer size:
41.1 KByte (default)
\------------------------------------------------------------ [ 3] local
192.168.100.34 port 5001 connected with 192.168.100.55 port 52536 [ ID]
Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [
3] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.004 ms 0/ 892 (0%) 0.086/
0.060/ 0.150/ 0.014 ms 89 pps [ 3] 0.00-10.00 sec T8(f)-PDF:
bin(w=10us):cnt(892)=7:160,8:143,9:181,10:159,11:242,12:6,16:1
(5/95%=7/11,Outliers=0,obl/obu=0/0)

Fedora 25: [root@hera iperf2-code]# iperf -s -u -e --udp-histogram=10u,10000
--realtime \------------------------------------------------------------
Server listening on UDP port 5001 with pid 16669 Receiving 1470 byte datagrams
UDP buffer size: 208 KByte (default)
\------------------------------------------------------------ [ 3] local
192.168.100.33 port 5001 connected with 192.168.100.55 port 35894 [ ID]
Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [
3] 0.00-9.99 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/ 892 (0%) 0.261/ 0.098/
0.319/ 0.021 ms 89 pps [ 3] 0.00-9.99 sec T8(f)-PDF:
bin(w=10us):cnt(892)=10:1,21:1,23:6,24:107,25:141,26:209,27:151,28:124,29:81,30:47,31:19,32:5
(5/95%=24/30,Outliers=0,obl/obu=0/0)

TX:

[root@rjm-clubhouse-28 rjmcmahon]# iperf -s -e -u --udp-histogram=10u,100000
--realtime \------------------------------------------------------------
Server listening on UDP port 5001 with pid 26016 Receiving 1470 byte datagrams
UDP buffer size: 208 KByte (default)
\------------------------------------------------------------ [ 3] local
192.168.100.55 port 5001 connected with 192.168.100.34 port 20343 [ ID]
Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [
3] 0.00-10.01 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/ 893 (0%) 0.094/
0.064/ 0.476/ 0.023 ms 89 pps [ 3] 0.00-10.01 sec T8(f)-PDF:
bin(w=10us):cnt(893)=7:12,8:38,9:291,10:311,11:198,12:28,13:10,14:1,15:1,42:1,45:1,48:1
(5/95%=8/11,Outliers=3,obl/obu=0/0) [ 4] local 192.168.100.55 port 5001
connected with 192.168.100.33 port 58391 [ 4] 0.00-10.00 sec 1.25 MBytes 1.05
Mbits/sec 0.013 ms 0/ 892 (0%) 0.079/ 0.048/ 0.127/ 0.009 ms 89 pps [ 4]
0.00-10.00 sec T8(f)-PDF:
bin(w=10us):cnt(892)=5:1,6:10,7:115,8:265,9:425,10:51,11:18,12:3,13:4
(5/95%=7/10,Outliers=0,obl/obu=0/0)

------
rurban
He forgot to say that Dave Miller brought in siphash for recent networking
hashes on Linux, which should bring down performance of the Linux networking
stack closer to Ruby/Python levels. About 20 slower. There's nothing slower
than this, whilst not improving security, but increased the accompanied
security theatre. It's a mess over there.

