
Line rate 10Gbit/s packet processing on FreeBSD with netmap - lrizzo
http://info.iet.unipi.it/~luigi/netmap/
======
muppetman
This is really very interesting.

I work with Juniper routers and switches all day and every day, the thing I
love about them is they're built on top of FreeBSD (now called JunOS, but it
still harks back to FreeBSD when you drop to the shell)

Something small and light and fast, running on FreeBSD with this added would
be perfect for customer CPE equipment or for non-essential routing tasks.

Very impressive, I look forward to seeing where this goes.

~~~
tptacek
The data path on a Juniper isn't passing mbufs around the FreeBSD IP stack, is
it? It's FreeBSD command-and-control, but my understanding is that the packet
processing itself might as well be its own rtos.

~~~
tryp
I don't know about juniper specifically but most multiport routers are built
on packet processing asics with content addressable storage in local ram and
built-in macs for each port. When a packet is encountered that requires more
complex interaction than the asic can handle it's kicked up to a CPU for
processing.

~~~
Nrsolis
This.

Juniper has always used a (heavily hacked) FreeBSD core running on an embedded
PC platform (Intel-based Routing Engine) for JUNOS that did all (or most) of
the protocol related stuff and left the actual forwarding of packets to a
specialized core of ASICs (The Packet Forwarding Engine).

This has changed in recent years with some lower-level protocols being handled
on the line-cards themselves on newer platforms. The vast majority of the
protocols still run on the Routing Engine though.

~~~
tptacek
Yeah, sorry, I was talking about the control plane stuff. Does Juniper use any
of the original FreeBSD IP stack for its control plane or exception path
stuff?

~~~
Nrsolis
I'm not really sure.

TBH, it's not really the limiting factor. The interconnect between the RE and
PFE is something like 100mbps.

------
sc68cal
As many have said on -CURRENT: Thanks for the great work. Do you plan on
getting this merged into the FreeBSD tree, and maintaining it?

~~~
lrizzo
yes to both questions. merging should start very soon.

~~~
kwis
Thanks again for the great work.

------
smutticus
This is impressive. How is the performance on different packet sizes? Most
real world loads would require far fewer than 14.88Mpps since most packets
would be equal to MTU. So I'm wondering if performance would increase or
decrease with packet size.

~~~
lrizzo
please look at the paper linked to the url, which defines the problem and the
metrics in more detail.

~~~
smutticus
Thanks. This is what I was looking for.

From the linked paper: _The most challenging situation is usually with the
shortest packet sizes, where per-packet costs are almost unavoidably dominant
over the memory copy costs. This justifies why we will run most of our tests
with 64 byte packets (60 + 4 for the Ethernet CRC). In netmap, in particular,
the per-byte system CPU cost is exactly zero because all data transfers are
performed by the NIC, and it is up to applications to access memory, if it
really needs to._

------
codepoet
Great! Maybe you could compare this with <http://www.ntop.org/PF_RING.html>
for Linux?

~~~
lrizzo
The "old" pf_ring is only meant for packet capture and involves packet copies,
so it is several times slower than netmap. There is a newer "Direct Network
Access" (DNA) version of pf_ring which avoids copies and has the same
performance of netmap, but is much more fragile because in DNA the userspace
program writes directly into the NIC registers and rings (so it can crash the
entire OS), whereas in netmap the NIC programming is filtered by system calls.

------
mateuszb
Very clever solution on stack bridging. +1

