
FreeBSD 10.0 on Ubiquiti EdgeRouter Lite - profquail
http://rtfm.net/FreeBSD/ERL/
======
rdl
In general for small hardware I go with two options.

Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of
hard to get in that config cheaply). You can get decent performance,
potentially hang a 5.8/2.4ghz radio on it too, and power it off a single
laptop USB port (200mA normal power use, and with stuff, still less than
500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current
commercial hardware in this space which is perfect. That should change
shortly.

Much beefier systems with "real" amounts of RAM and flash. I generally skip
the Broadcom/RPi/etc. and go straight to an Atom or i3, but that's just for
convenience, since I only need a small number of them. There's probably
something more reasonable to do in the embedded space, but I'm indifferent on
qty 1 devices between $500 and $200, especially if I already have most of the
components for the $500 thing lying around, and it saves me an hour or two
getting it working.

There are a lot of things missing from widely available commercial hardware in
this space right now.

~~~
stephen_g
I've been looking at Intel's DPDK recently [1], and it looks really
interesting for really fast packet processing in the 'beefier' kind of systems
you describe.

It would be a bit higher cost, but could mean you could make a router like the
EdgeRouter but with good performance without the proprietary accelerator (I
believe that Intel's drivers are open source).

1\. Data Plane Development Kit, which lets you skip the kernel IP stack (which
takes thousands of CPU cycles to process) and do packet processing in userland
taking just tens to hundreds of cycles per packet.
[http://dpdk.org/](http://dpdk.org/)

~~~
pktgen
> 1\. Data Plane Development Kit, which lets you skip the kernel IP stack
> (which takes thousands of CPU cycles to process) and do packet processing in
> userland taking just tens to hundreds of cycles per packet.
> [http://dpdk.org/](http://dpdk.org/)

I wish OS developers saw this as a problem. There is no reason kernel stacks
should be so slow for tasks where all processing is done in the kernel. (For
packets destined to userspace, you've got the syscall overhead to deal with.)

I recently tested the Linux network stack's PPS performance with an Intel X520
10GbE NIC. I used Debian testing, with the 3.12 kernel. My destination machine
was an i7-3930K at stock speed. I wrote a simple kernel module adding an
NF_IP_PRE_ROUTING hook returning NF_DROP with no processing, which would be
the simplest possible code path. For a packet generator, I used another older
machine with another X520, using the "pfsend" tool included with PF_RING, and
the card in PF_RING DNA mode. That was easily able to saturate the link at
line rate (14.8M PPS).

The result: the kernel was only able to sustain about 2.8M PPS.

I then loaded the DNA driver on the destination machine, used the included
"pfcount" tool, and no packet drops - it was receiving the full 14M PPS.

I tested DPDK recently and had similar results.

I also modified the Linux ixgbe driver ixgbe_clean_rx_irq() function, and
added a step in between the "fetch packets from RX ring" and "put packet in
SKB and send to network stack" functions. Even when I added a bunch of useless
comparisons for each packet, I was able to get ~12-13M PPS. I could get line
rate by just dropping and not doing any processing.

~~~
rdl
I wonder if CloudFlare does any of this.

~~~
pktgen
They do. They use Solarflare's OpenOnload: [http://blog.cloudflare.com/a-tour-
inside-cloudflares-latest-...](http://blog.cloudflare.com/a-tour-inside-
cloudflares-latest-generation-servers)

~~~
rdl
Those look nice, but presumably Intel NICs would be an order of magnitude
cheaper, especially if built into the chipset.

~~~
pktgen
Definitely. I am not sure what metrics Solarflare beat others (presumably
including Intel) on, besides software support (OpenOnload implements the BSD
sockets API and doesn't require software changes, while DPDK and others
implement their own). In my testing, the X520s have been capable of both
sending and receiving at line-rate.

It's possible the Solarflare NICs could be better at some metric like latency,
by a matter of microseconds (only speculating here), but I can't see how
they'd consider that worth the extra money for HTTP servers.

------
zx2c4
I've been running Gentoo Linux on my EdgeRouter Lite for a long time now.
<[https://wiki.gentoo.org/wiki/MIPS/ERLite-3>](https://wiki.gentoo.org/wiki/MIPS/ERLite-3>)

I must say: the ERL is terrible hardware.

You can get some performance out of it if you're using Ubiquiti's kernel,
because it ships with a series of highly specialized _proprietary_ modules
that offload packet handling to hardware accelerated modules, for their custom
_proprietary_ network stack. But their kernel is old and shitty, and if you
want to do anything interesting with the ERL, you have to build your own
kernel.

So for any mildly interesting usage, it fails considerably hard.

And even if you do opt to use their closed-source binary blob kernel modules
and software stack, as soon as you try to do any sort of advanced networking,
you run up against the limits of their acceleration modules, and the OS is
forced to fall back to standard non-accelerated mode, and performance drops.
Try using PPP, any interesting iptables, MPTCP, OpenVPN, interesting routing
tables, or anything cool: performance drops.

I would be very happy to see some small networking hardware with multiple
independent Ethernet controllers pop up on the market for a decent price;
every other month or so my disappointment with the ERL moves me to re-research
this. Unfortunately, the EdgeRouter Lite is not the hardware anybody wants for
interesting low-cost networking.

~~~
purple_horse
I also do not like Vyatta's weirdness. It looks like Linux, acts like Linux,
but then you have to use all of the awful Vyatta commands to actually do
anything.

~~~
jauer
If you are expecting a Linux distribution then yeah, it is weird. If you are
expecting a network appliance then it's par for the course and you are
grateful that you can drop to shell to do other things. JunOS is pretty much
the same way.

------
jlgaddis
Related: I picked up a couple of Neoware CA21 thin clients on eBay real cheap
recently. 800 MHz, 512 MB RAM, and 1 GB disk-on-module. I built customized
(NanoBSD) images and run them read-only off of a flash drive. I connected a
Garmin GPS 18 LVC to one and have my own stratum 1 NTP server for < $100.

~~~
zx2c4
Unfortunately, only one shitty Ethernet port. I'd love to find fast hardware
with multiple independent gigabit ports for an affordable price.

~~~
gnoway
[http://www.pcengines.ch/apu.htm](http://www.pcengines.ch/apu.htm) ?

Presumably this will actually be released at some point soon.

Also the soekris hardware looks nice, but it's pretty expensive.

~~~
zx2c4
/me salivates

Unfortunately by the time it's released, the hardware will already be a bit
old. But nonetheless, the specs are certainly sufficient for most cool things.

------
zdw
Similar, but not as mature support is being worked on in OpenBSD as well:
[http://www.openbsd.org/octeon.html](http://www.openbsd.org/octeon.html)

~~~
hiphopyo
OpenBSD is amazing.

~~~
ubercore
I think the "not as mature" comment was about the support for it, not a
comment on OpenBSD.

------
gnoway
Note that UBNT claims line rate throughput and provides proof via a
commissioned Tolly report (no idea on their legitimacy).

The fbsd image yields ~250mbps between two gigabit hosts.

I've been eyeing one of these for a while and putting fbsd on it interests me,
but not if it means a huge peformance drop (even if I wouldn't really miss
it).

~~~
profquail
Assuming UBNT is correct and the device supports line-rate throughput, I don't
think there's any reason why FreeBSD couldn't easily support that.

The freebsd-mips mailing list already has a few threads discussing this; if
you want to know why the router isn't achieving line-rate speeds "out of the
box", you'll probably get the best answer there (and maybe even someone who
knows how to fix the problem):

* [http://lists.freebsd.org/pipermail/freebsd-mips/2014-January...](http://lists.freebsd.org/pipermail/freebsd-mips/2014-January/003263.html)

* [http://lists.freebsd.org/pipermail/freebsd-mips/2014-Februar...](http://lists.freebsd.org/pipermail/freebsd-mips/2014-February/thread.html)

If you don't get a suitable answer there, you can also try freebsd-net:

* [http://lists.freebsd.org/pipermail/freebsd-net/](http://lists.freebsd.org/pipermail/freebsd-net/)

~~~
pktgen
UBNT is correct, but only for the hardware fast path. The driver for that
appears to be proprietary.

~~~
profquail
I think you're right about the driver. There's some more info about that here:
[https://wiki.freebsd.org/FreeBSD/mips/Octeon](https://wiki.freebsd.org/FreeBSD/mips/Octeon)

------
justincormack
The downside of this right now is that there are no binary packages for mips64
and there is no space on the device to store a ports tree so you need to cross
compile everything.

I am running gentoo on mine at the minute, I did have FreeBSD on it for a bit.

I tried a larger flash drive but the driver doesn't like it and won't mount
the root partition, need to find one it likes. Or use nfs/smb but that is a
pain.

~~~
zx2c4
I successfully connected a USB hard drive to mine. If you're interested
enough, email me and I'll go look up which inexpensive model of enclosure I'm
using. The bootloader is indeed finicky but I guess I got lucky enough to have
a HDD enclosure that just worked.

~~~
justincormack
I must have some lying around, will give them a go. The bootloader works with
my USB stick, but Linux doesn't like it. Will see if FreeBSD does or not.

------
blutack
As an easier to obtain alternative to minicom, screen can also handle serial
port communications quite happily for basic usecases (like this one).

    
    
      screen /dev/ttyUSB0 115200 # Linux-centric example

~~~
profquail
On FreeBSD, _screen_ is available in the _sysutils /screen_ port:
[http://www.freshports.org/sysutils/screen/](http://www.freshports.org/sysutils/screen/)

------
helper
Neat. I've been using an ERLite for the past ~6 months. It does its job, but
I'm not super excited by it. I find the Vyatta interface to be a little
annoying (most enterprise network gear I've worked with just copied Cisco's
CLI exactly, Vyatta is just different enough to be surprising).

I'm thinking about swapping it out for a Mirabox[1] I have laying around. Its
an ARMv7 with dual Gigabit interfaces and 1GB of memory. With the 3.14 kernel
the SoC is quite well supported. Now if only I could run pfSense on it.

[1] [https://www.globalscaletechnologies.com/p-58-mirabox-
develop...](https://www.globalscaletechnologies.com/p-58-mirabox-development-
kit.aspx)

~~~
justincormack
Do you know if it can drive the interfaces at anything near gigabit? That
seems to be a problem with a lot of ARM chipsets.

------
Fasebook
Anyone know anything about the base-band chip and it's OS?

~~~
justincormack
base-band? There is no wireless.

