
What Linux can learn from Solaris performance and vice-versa - deirdres
http://www.slideshare.net/brendangregg/what-linux-can-learn-from-solaris-performance-and-viceversa
======
sharms
This is a topic that I would love to see brought up more, and I think there is
a lot Linux can learn from Solaris. As someone who was only exposed to Solaris
in the last 4 years (using Linux since '96), I have become increasingly
interested in the seemingly unimportant features which have been incredibly
useful.

Zones have a distinct maturity and robustness over LxC (to be expected), ZFS
is the filesystem I wish we could have, dtrace isn't a bolt-on etc.

It would be nice to see it fully open-sourced again as I think the one thing
it has working against it is a smaller community and it lacks the same size
army of driver authors.

~~~
bcantrill
As rasur mentions, illumos has inherited the Solaris heritage -- and added
quite a bit to it besides.[1][2] There are now several vibrant variants of
illumos, including SmartOS[3][4], OmniOS[5][6] and DelphixOS[7] -- all of
which are open source, and all of which feature advancements in the core
differentiators of the system including ZFS, DTrace and zones. Because we
collectively have been entirely separate from our Solaris legacy for over
three years now (and because we are entirely open source where Solaris is
entirely closed source), the differences are actually substantial -- and I
don't really think of our system as "Solaris" any more than I think of it as
"System V" or "7th Edition" or anything else in our long and complicated
ancestry...

[1]
[http://www.youtube.com/watch?v=-zRN7XLCRhc](http://www.youtube.com/watch?v=-zRN7XLCRhc)

[2]
[http://www.youtube.com/watch?v=7YN6_eRIWWc](http://www.youtube.com/watch?v=7YN6_eRIWWc)

[3] [http://smartos.org/](http://smartos.org/)

[4] [https://github.com/joyent/illumos-
joyent](https://github.com/joyent/illumos-joyent)

[5] [http://omnios.omniti.com/](http://omnios.omniti.com/)

[6] [http://omnios.omniti.com/browse.php](http://omnios.omniti.com/browse.php)

[7] [https://github.com/delphix/delphix-
os](https://github.com/delphix/delphix-os)

~~~
angersock
So, to a newcomer, what are the things that these flavors offer over, say,
stock Debian?

~~~
thelambentonion
While I haven't had a chance to use illumos in any of my projects, the Service
Management Facility (SMF)[0] and Fault Manager Daemon (FMD)[1] are two big
reasons why I'm looking at switching from FreeBSD to an illumos variant for my
home server.

It's been a little while since I've used Debian, but I don't believe any
systems comparable to SMF or FMD have been implemented[2].

[0] [https://www.illumos.org/man/5/smf](https://www.illumos.org/man/5/smf)

[1] [https://www.illumos.org/man/1M/fmd](https://www.illumos.org/man/1M/fmd)

[2] I'm not sure how well systemd compares to SMF, and I'm mostly interested
in FMD/SMF due to how well it can handle ZFS fault management.

------
nailer
'Linux might get DTrace-like capabilities via SystemTap'

Might? SystemTap's been stable & out of the box on RHEL 5 for years. The
extensive list of default tap points are supported on production by RH. When I
was doing systems stuff 5 years ago we used it for double checking TCP socket
options that weren't normally exposed to userspace tools.

Dtrace can do some awesome stuff stap can't, like trace something from
userspace down to kernel space. But systemtap does provide live
instrumentation on running kernels and has for years.

PS: Just saw p93. Sun got in trouble for the obvious "Setting up SystemTap is
difficult on distros that don't maintain, support, use or like SystemTap"
years ago. This is repeated on p 93. Stop it, this makes Joyent look stupid,
you're better than this.

And p100. Re: 'several million dollar E10K' performance? Red Hat were
replacing them with Xeon's by 2003 (with Sun's old sales staff) and crushing
them. Though that was SPARC's fault not Solaris's.

~~~
brendangregg
It would be great if all Linux distributions had SystemTap out-of-the-box like
RHEL. Or something with a sufficient feature set to solve problems (which may
become ktap, or perf_events, or lttng). However, most of the Linux performance
issues I'm debugging are on Ubuntu systems. So the question remains: when will
Linux-at-large get this capability?

Red Hat can certainly say that SystemTap is obviously a priority on their OS,
and they can only make best effort for others. I said this in the talk.
Indeed, a working SystemTap should be an incentive to run RHEL. It is a
compelling argument -- the benefits of a working SystemTap are enormous (which
I also explained in the talk). Sun made basically the same argument for
switching to Solaris: a working DTrace.

Although, I'm not sure that using SystemTap is entirely safe for production
yet, which I said was the most important priority. I notice that this related
kernel panic is still not fixed:

[http://sourceware.org/bugzilla/show_bug.cgi?id=2725](http://sourceware.org/bugzilla/show_bug.cgi?id=2725)

Although some recent and promising progress has happened in the past few
months, as noted by the bug! I only care about this bug because I sometimes
trace large subsets of the kernel (eg, all functions in a module or driver),
and I think I've hit the same related issue. I don't actually need to trace
everything.

You said that DTrace can trace from userspace to kernel, which SystemTap
can't. I'm not sure if there's a specific case you're referring to, but
SystemTap nowadays can indeed trace userspace to kernel (via uprobes), and a
lot of work has been happening to get SystemTap userspace tapsets to work.

As for setting up SystemTap being difficult on distros that don't support or
maintain SystemTap. Well, almost all of my customers run Ubuntu. If they hit
perf issues, do I convince them to switch to RHEL? (The approach that Sun made
for DTrace.) Or is this a problem to be taking up with the other Linux distro
maintainers? Which goes back to my original point: when will Linux get this?
:-)

~~~
ema_rocca
Is it problematic to install SystemTap on Ubuntu? On Debian systems it's
straightforward.

apt-get install systemtap linux-image-amd64-dbg

~~~
_delirium
It's not too much harder, but requires a non-default respository, so mildly
harder. Quasi-official instructions here:
[https://wiki.ubuntu.com/Kernel/Systemtap](https://wiki.ubuntu.com/Kernel/Systemtap)

The difference is that, as of the past few releases, Ubuntu doesn't ship a
-dbg kernel package in the main repository like Debian does. Instead they've
come up with a new system to ship 'debug symbol' variants of _all_ packages,
with a -dbgsym suffix. But they ship them in a separate repository that isn't
enabled by default, I guess to avoid confusing end-users (ddebs.ubuntu.com).
So you need to configure that repository, install its key, then pull linux-
image-$(uname -r)-dbgsym. I guess you might also want to pull the debug-symbol
versions of the main libraries as well.

------
the_mitsuhiko
The sad part is that both dtrace and ZFS originally did not end up in the
kernel purely because of licensing reasons :-(

~~~
throwaway2048
There is more to it than that, the CDDL was intentionally created by Sun to be
GPL incompatable, in a bid to create their own linux-like ecosystem with
OpenSolaris.

The incompatable nature of the license was so linux could not merge things
like ZFS and dtrace.

it was yet more Sun licensing games that only hurt adoption in the end, like
their Java shenanagins.

Summing it up as "aww gee shucks licensing problems have struck again" ignores
the fact that such crap was an intentional power-play from sun.

~~~
lucian1900
After an intentional power play from GNU. Let's not absolve them of blame.

A clash of politics had a sad effect on actual technology.

------
patrickg_zill
One thing I noticed about Solaris - it handles low-memory situations (like
15.5G out of 16GB being used) and swapping, very very well.

If Linux has enough RAM, it will outperform Solaris, but, it is quicker to get
bogged down if there is swapping.

(These observations are from about Centos 5.x , so, perhaps they are no longer
valid, however.)

~~~
GFK_of_xmaspast
I find myself having to run with ulimit -v #### in a term on linux if I think
I might start eating memory because I'm sick of having to craw under my desk
to power cycle.

~~~
ithkuil
why not turn off swap entirely?

------
apaprocki
What surprised me most is that SPARC is only ~2x slower after all these years
(3.6ghz T5 / Sol11) ;)

    
    
        $ time perl -e 'for ($i = 0; $i < 100_000_000; $i++) { $s = "SCaLE12x" }'
        
        real    0m33.59s
        user    0m33.57s
        sys     0m0.00s

~~~
twoodfin
I know multithread is the T5 sweet spot (to put it mildly), but my 2012 Retina
MacBook Pro hits 10s and cost ~$2,800. The cheapest T5 I can find on Oracle's
site starts at around $28,000.

I'm only jealous because I'd really like my employer to buy a T5 for compiling
and running our software on Solaris SPARC, but unfortunately I don't think
it's justified given the state of the platform.

------
jackgavigan
How can you talk about Solaris performance without mentioning dispadmin?

~~~
brendangregg
Linux has CPU scheduler tunables via CFS/cgroups. dispadmin allows fine tuning
of the dispatcher table, but I've never seen that provide significant or
consistent wins for production (and I've had use cases where it seemed
possible in theory, and dtrace'd dispatcher queue latency, but dispadmin
didn't really help.) I might mention dispadmin in a 5 day talk. :)

------
binarycrusader
It seems like futexes are really just an alternative interface made to build
pthread mutexes on top of. In addition, they may only be faster on Linux
because of implementation issues. It doesn't seem worth including for
comparison.

Also, I'm confused by the claim that Solaris doesn't have the vfsstat utility;
yes, it doesn't have that utility, but it does have "fsstat" which provides
VFS-level statistics. So it'd be helpful to see a more detailed comparison.

------
angry_octet
I detest slideshare. The way they try to monetize and track me by making me
log in to download the PDF. The dumb way they show the slides on a third of
the screen and fill the rest with crud.

So if you really don't want your slides read, put them on slideshare.

~~~
brendangregg
There is a direct PDF link here:

[http://www.brendangregg.com/linuxperf.html](http://www.brendangregg.com/linuxperf.html)

------
deirdres
Video now available: [http://www.joyent.com/blog/what-linux-can-learn-from-
solaris...](http://www.joyent.com/blog/what-linux-can-learn-from-solaris-
performance-and-vice-versa)

------
vidoc
Sure yeah, Solaris is full of great technologies, it is undeniable, and DTrace
is an awesome and unique one.

Now there's one thing those guys can learn from Linux: modesty.

------
bingaling
Looking forward to the video!

------
GFK_of_xmaspast
The last bullet in slide 83 isn't exactly a selling point.

------
lazyloop
Just yesterday Perl has once again been declared dead on Hacker News, and here
we have it happily strutting along in a keynote about a bleeding edge topic.
:)

