
Unscrewed: A Story About OpenBSD - netten
http://www.skeptech.org/blog/2013/01/13/unscrewed-a-story-about-openbsd/
======
nvahalik
I'll never forget this time in college when I had set up an OpenBSD router in
our apartment and came home one weekend to find that I was unable to log into
it via SSH.

Actually, it was timing out.

So I hooked up a monitor and a keyboard to this old old P1/133 with 32MB of
ram that had been taken out of a dumpster to be our router to see what was
going on.

A few months prior to this I had been playing around with rrd and started
generating traffic graphs so that I could get pretty pretty pictures of our
traffic (and to figure out when one of my roommates was streaming his porn).

I had set the cron jobs to run every 10 minutes. Initially the data sets were
so small that this wasn't an issue, but over time the database got larger and
suddenly the cron job was taking 10 minutes and 10 seconds, and then 10
minutes and 20 seconds... on and on until finally cron jobs were running over
cron jobs.

So after about 2 minutes waiting to get logged in, I finally get the load
average for the system and it was somewhere around 146.20. After another
couple of minutes, I was able to disable the cron jobs and kill off rrd
graphs. In another 10 minutes, the load average was back to < .70.

All this time it was passing traffic without a hiccup and I believe it had
another 100+ days of uptime after this before I moved out. What a box. I miss
that router.

~~~
profeta
ah, nobody ever forgets their first lock file.

------
SwellJoe
I've always (or, for many years) wondered why this wasn't more common. Every
network I've ever built has been routed by Linux or a BSD, though I've
maintained a few with Cisco routers. Partly because it's what I know, and
partly because there's never enough in the budget to to everything the client
and I wanted to do with Cisco or even Juniper gear.

Now, admittedly, I've never worked on big networks. But, I know what I like,
and I know working on a Cisco feels like 1996, and that working on a full-
blown modern UNIX/Linux system feels like home. And, fast network gear _is_
available for Linux and the BSDs. 10Gb interfaces that work with Linux/BSD are
not difficult to find, and 1Gb interfaces (sometimes several of them) have
been _standard_ on every server and barebones rackmount on the market for a
decade.

I understand there are special purpose boxes that do things no off-the-shelf
commodity box can do (I don't know enough about high end networking to know
where to draw that line), but I honestly doubt 95% of business networks need
them. And, yet, a huge percentage of business networks that have real routers
(rather than little wifi routers from LinkSys/NetGear or whatever) have a
Cisco of some sort, even when they don't have anyone on staff who knows how to
work the thing. I've walked into several businesses that had expertise in
house for everything _except_ their router, which is also weird...why pick a
Cisco router for your little ~100-1000 node network, if you've got Linux or
BSD guys hanging out in the server room every day? They'd call someone and sit
around waiting when their network was acting up.

I learned how to use Cisco routers just so I'd be able to replace them easily
(enough knowledge to read the routing tables, firewall rules, etc., but not
enough to do much useful), when I found situations like that.

------
aexaey
Regarding first paragraph of the story (redundant pair or routers crashing at
once), it worth mentioning one possible neat trick to avoid this: When having
a redundant pair of devices (routers, etc.), make sure such pair is comprised
of devices made by _different_ vendors - thinking here being that if there is
a bug that triggers router crash, it's unlikely that two different vendors
would be vulnerable to the same one.

Or with a generous budget, you can do what BT have done with their 21CN
project: For each macro-block (access, aggregation, core), have two redundant
_networks_ each one built by a different set of vendors.

~~~
_yy
No such thing as session synchronization for different vendors. You also need
to configure everything for both of these vendors, which may or may not have
equal feature sets.

~~~
jerf
It's hard to have independence without independence. You want separate, you're
going to have to deal with separate.

~~~
_yy
Yeah but no sane organization is going to do that. It would create so much
headache.

~~~
jerf
A sane organization will do it if the benefits outweigh the risks. An
organization with sufficient risks will chose it. Most won't, because the
risks (specifically, hard cash) won't be outweighed by the benefits.

But that's incidental to what was my real point, which is that if you do
_want_ a separate network, it has to be _separate_. Tie your two "separate"
networks together with integration and you're returning single-points-of-
failure back into the mix. Granted, practicality may dictate a couple of
those... integrated authentication comes to mind... but you don't _want_ the
two separate networks to be all slickly and smoothly integrated for the most
part.

------
scurvy
A couple of elephants in the room:

Why on earth were you running a hosting business with Cisco 2950s? Those are
wiring closet desktop switches with tiny buffers. That alone probably caused a
lot of pain. You didn't know about it, but they did.

Second, the Juniper J4350 is a software router based on BSD. There's no
special hardware in there like ASICs. It's just a PC.

JunOS (historically) is based on BSD. They're moving to Linux now.

I love OpenBSD for small projects too, but let's admit that there was some bad
logic in this decision making process. This could be called "how the mx10
saved my ass because I went with undocumented, unproven open source project
that has been on github for 3 days" if the technologies were reversed.

~~~
mioelnir
> JunOS (historically) is based on BSD. They're moving to Linux now.

Do you have details on this? Given how active they are in the FreeBSD
community, I would have thought to have heard about that already.

~~~
nemith
Depending on how you look at it some platforms are already running Linux.

QFX5100 runs a Linux KVM hypervisor with the traditional FreeBSD based Junos
running as a guest. Same with the new x86 based SRX (SRX1500). There are more
moving to this model.

The x86 based SRX is the most interesting with the forwarding daemons running
on the hypervisor OS and the Junos based VM is just for management and
protocols. It's not that hard to see that management and protocols can be
easily ported.

There are more details that are locked behind an NDA.

------
ubercow
What's the difference between OpenBSD and FreeBSD. I've heard great networking
things about both, but never really seen them compared side by side.

~~~
gknoy
According to Wikipedia [1], OpenBSD optimizes for maximum security, whereas
FreeBSD optimizes for performance, and NetBSD optimizes on portability.

1:
[https://en.wikipedia.org/wiki/Comparison_of_BSD_operating_sy...](https://en.wikipedia.org/wiki/Comparison_of_BSD_operating_systems)

~~~
Cyberdog
As might be expected from a description so terse, this is a bit of a
generalization. I think my preferred explanation is that FreeBSD aims to be a
more usable general purpose operating system, whereas OpenBSD is more
conservative, preferring to grow at its own pace and in its own directions
which may not necessarily be the state of the BSD art in terms of user
friendliness. A great example of this is ZFS; no free or paid OS outside of
Solaris supports ZFS to the extent that FreeBSD does, but if you want to use
ZFS with OpenBSD, you're outta luck - it's not even on the roadmap. Last I
checked, OpenBSD had no interest in moving from GCC to Clang/LLVM either.

That being said, both OSes are free, and so is VirtualBox, so why not spin up
a couple virtual boxes and give them both a try when you have the time?

------
bitwize
About... two decades ago I set up an OpenBSD box for my boss to use as a basic
web host to pass stuff to clients.

The only time it failed -- the ONLY time -- was due to a full hard disk. Why
was the hard disk full? Silly me, I forgot to implement some sort of logging
discipline and a couple years' worth of logs of attempted exploits (mainly
Windows NT exploits) glancing harmlessly off OpenBSD's hull had filled the
disk.

------
mey
From the article "So there I was, right out on the razors-edge, implementing
production networks with an untested OS on brand new hardware. And oh how it
failed."

Doing Enterprise IT Arch work, I turned down more than one hot new system and
put others through significant lab work, to avoid this fun. It's hard, it's
expensive and it's why I wish more hardware was open.

------
wolf550e
Can that hardware keep up with crypto bitrate and firewall rules, on multiple
10gb links?

~~~
corysama
The Snabb Switch crew are starting to work on support for basic handling of
packets on 100G NICs. However...

> With the new CPUs tending to increase the core count while the CPU frequency
> goes down, we'll get less and less "CPU cycles per packet". For example
> E5-2699v3 45M is 18 cores@2.3GHz. 10Gbps with 256 byte packets is achieved
> with 4.53Mpps, this makes ~500 clocks per packet on this CPU. Now make this
> 100Gbps and you get 50 clocks. You get this budget to receive, process and
> send the packet. 50 sounds pretty low to me.

Numbers refer to using only a single core. That particular CPU has 18 cores
and 36 hardware threads.

[https://groups.google.com/forum/m/#!topic/snabb-
devel/pDd_uB...](https://groups.google.com/forum/m/#!topic/snabb-
devel/pDd_uBXszrY)

~~~
richardwhiuk
Normally the solution is to have the NIC give you multiple receive queues and
then tie each queue to a different core, meaning you get a packet on each core
every time you clock them off the NIC. Increasing CPU cores gives you a
massive win when you do that.

(On a hardware note, increasing cores and decreasing frequency makes the CPU
look more like an ASIC :) )

------
joslin01
I enjoyed the story, brother.

------
lmm
Commodity hardware and general-purpose systems win every time.

Honestly I wonder, who the hell buys these overpriced networking appliances?
Similarly with expensive proprietary databases, or ESB systems. Are they just
leftovers of a different time (note article is from 2007)? Is it a case of
management buying from the salesman who bought them a nice dinner? Some place
where these things generate actual value that I'm missing?

~~~
detaro
People who want to have support by a large vendor. If your DIY-setup fails you
can't blame a vendor. In theory using the same kind of appliance as everybody
else means you can easily find an networking guy to work with it, people who
are good at both traditional networking gear and BSD/Linux are harder to find.
If a dedicated box fails you can just have someone with some experience plop a
new one in, reapply the configuration and be done. Correctly replicating a DIY
setup is harder, especially if the person building it didn't do a perfect job
at documenting it.

Appliances promise all sort of fancy features that might be hard to exactly
replicate yourself (if you actually need them often is a different question,
but often networking isn't exactly in a position to say "no" to such
requests). Although there is more and more a trend of manufacturers also
offering the software for virtual environments, many appliances are
x86-servers anyways.

\+ as the article mentions all those cases where you need specialised
hardware, e.g. routing/switching at high speeds, or working in special
environments.

~~~
EvanAnderson
I've heard the "blame a vendor" argument again and again in my life, but I
have yet to see this "blame" ever occur. Granted, I'm dealing with small and
medium-sized companies, but I've never seen any kind of tangible financial
outcome from "blaming" a vendor (other than that vendor not getting future
sales).

~~~
profeta
"nobody gets fired for buying IBM"

[https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt](https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt)

Everyone who says anything like "blame the vendor", have just drank too much
marketing cool-aid.

