
The limits of open source with Illumos and OmniOS - jsnell
https://utcc.utoronto.ca/~cks/space/blog/solaris/IllumosSupportLimits
======
bahamat
As has been pointed out on Twitter,

> this is funny because his main beef is actually due to systemic problems
> inside Intel, not Illumos or FOSS

[https://twitter.com/kevinbowling1/status/714352429333086208](https://twitter.com/kevinbowling1/status/714352429333086208)

~~~
kev009
Since I can elaborate more on here vs twitter, I have a senior engineer
spending 100% of his time fixing Intel drivers at my company for coming up on
2 years. The FreeBSD driver is somewhat related to the Linux driver but Intel
has obligations to put out a copyfree implementation. I believe this was then
ported to Illumos a couple years ago.

ixgbe works relatively well on Linux and FreeBSD right now, but there are
still occasionally surprises and it is less efficient than other options.
What's astonishing is that you can get better HW with _much_ better drivers
from other companies for cheaper. The Chelsio t520-so-cr is unequivocally
better than the intel x540 and costs less. Chelsio, Mellanox, SolarFlare are
all good choices for Linux, FreeBSD. I think Chelsio has a Solarish driver for
Illumos, not sure about the others.

Inside Intel, the Windows team, Linux team, FreeBSD team do not talk to each
other. They appear to not be able to talk to the HW team either. Several large
and influential companies have been trying to force Intel to clean up their
FreeBSD drivers. They have taken action, and that action has been pretty
disappointing. The Linux driver and commit logs are also illuminating since
that would presumably have massive market share.

Intel's 40g parts have been fraught with issues at the HW level. The drivers
were barely able to outperform 10g at release. This kind of slop is not
normal. It should not be rewarded in the market. I had an uphill battle
convincing old timers that Intel NICs went so far down hill from the good old
days, but after a lot of analysis we have totally written them off for the
next two years.

~~~
JoeAltmaier
Is it truly hardware? I had to deal with radios for years, and the vendor-
supplied driver was always poor. It came from some SOM company, who got it
from the radio designer. It was always a demo driver, intended to show off the
chip features but no effort put into performance or error recovery.

It seemed nobody in the 'chain of custody' of a driver had any incentive to
make it work well, in a commercial setting. At best, it was consumer-quality.
By that I mean, it worked until it didn't. For a radio, it meant if roaming
jammed up then just take the radio dongle out and put it in again. Which in a
commercial device (like a forklift touchpad) which had the radio sealed behind
a panel, it was junk.

So I had to fix features, performance, bugs, timing, power management, the
works. E.g. to get a radio driver fit for a WalMart distribution center
forklift going 15mph, it had to roam in milliseconds and choose between 60 APs
in radio range. And run for a 12 hour shift without recharging. The chipmaker
driver was never, ever good enough.

~~~
kev009
Dollar for dollar, the Chelsio and Mellanox HW is better in terms of features.
Better drivers and vendor support at lower purchase price make it a no
brainer. I don't know about SolarFlare pricing but it is also better HW.

igb and ixgbe HW seem to be fair, but you can go look at the HW errata to
judge for yourself. Intel had to recall the XL710 due to silicon issues. They
also had a firmware incident that fundamentally changes the driver interface,
so a particular driver will not work between different FW revs.

------
cyphar
The solution is obvious: pay someone to do the work. That's how 80% of Linux
kernel development is done. Just because it's free software doesn't mean that
you shouldn't spend money to make it better.

------
cm3
Is ixgbe really that much worse than Chelsio's?

~~~
kev009
In a word: yes

~~~
cm3
Hardware as well?

~~~
kev009
Yes, but hopefully anyone doing this for real will compare whitepapers and
errata for yourself, then lab test them. The T5 ASIC is in a different class
than intel's parts, it can be used as several types of offload engine if that
seems like a good fit for things like iSCSI, or it can be used as a very solid
stateless offload NIC with tons of VFs for things like Netmap, zones/jails, or
HW virtualization. Most people don't push NICs very hard, so it doesn't matter
that much. If you do, you probably already have a strong opinion (i.e. glance
at Netflix OpenConnect build docs).

~~~
cm3
I was planning 10G networking in my home lan, so I guess I'll eye Chelsio NIC
then. Hope they're not too expensive.

~~~
kev009
Typically cheaper than Intel I bought from Netgate for my personal colo
machine [http://store.netgate.com/Chelsio/T520-SO-
CR.aspx](http://store.netgate.com/Chelsio/T520-SO-CR.aspx)

~~~
cm3
Doesn't Chelsio offer copper 10G port NICs? 10G non-fiber (aka Cat.7) has been
my plan.

~~~
kev009
They do, but twisted pair uses more power. You should use Twinax copper cables
with the SFP NICs and switches if you don't want fiber.

