
Joyent Open Sources SmartOS: Zones, ZFS, DTrace and KVM - timf
http://smartos.org/
======
mrb
Wow this is huge. Having an OS be able to run both ZFS and KVM enables
fantastic things. For example you can store each virtual machine disk template
on a dedicated ZFS filesystem, and use "zfs clone" to rapidly deploy VMs
(instead of using the KVM-level support for base disk images "qemu-img create
-b ..."), as well as "zfs snapshot", "zfs revert", etc. The main advantage
being that these clones and snapshots are possible while using the simple and
fast "raw" KVM disk image format instead of the notoriously slower "qcow2"
format that was, until today, the best format supporting base images and
snapshots.

I, for one, have been wanting to use ZFS specifically like that for a while.
This does not compare at all to running ZFS on, say, an NFS server or iSCSI
SAN serving data to a server running KVM (ZFS data integrity is only verified
remotely on the storage server, it is slower, etc).

Who ported KVM to Illumos? I know some old version of QEMU was running on
Solaris at some point in the past, but I had no idea they had a full blown KVM
port.

~~~
bcantrill
KVM was ported to illumos by Max Bruning, Robert Mustacchi and me. Details are
here: <http://t.co/knW1UIJ>

~~~
slillibri
Kind of an off topic rant, but can we at least avoid url shorteners that point
to other url shorteners? Here is the original link
[http://www.slideshare.net/bcantrill/experiences-porting-
kvm-...](http://www.slideshare.net/bcantrill/experiences-porting-kvm-to-
smartos)

~~~
mambodog
Honestly, no one should be using URL shorteners on HN. Use a footnote[1] if
necessary.

[1] <http://www.google.com/search?q=hi,+this+is+a+footnote>

~~~
RexRollman
I like the footnote form. I use that in e-mails as well.

------
nyellin
To clarify, Joyent open sourced SmartOS. Zones, ZFS, DTrace, and KVM are
maintained by other companies and have been open source for ages.

Edit: That's not to imply Joyent is freeloading. There is another post on HN
with an example of the awesome work they're doing:
<http://news.ycombinator.com/item?id=2887092>

------
strlen
This is really great. Joyent has been running this internally for years. The
combination is just great: ZFS, DTrace and KVM speak for themselves, but
another great ingredient is their use of the NetBSD userland (in place of
legacy Solaris one).

Linux is great, but a monoculture benefits no one.

------
rednaught
I'm having a difficult time wrapping my head around this. If Joyent wanted to
stick with a UNIX then why not FreeBSD and help implement KVM there? Wasn't
XEN already available for Dom0? I just don't see much mindshare in
Illumos/SmartOS for keeping up with current server hardware drivers. I really
would have liked to see one of the existing BSDs benefit from the time they've
put into this project.

Obviously Linux already has KVM, and there are new implmentations of DTrace
and ZFS. Btrfs is not too many more kernel releases away from being considered
stable(fsck being a glaring problem) and if you want container-based
virtualization you've got LXC(some prefer OpenVZ).

With that said, who is their target audience?

~~~
dap
You're suggesting Joyent should have finished porting DTrace, ZFS, and Zones
to FreeBSD, and then port KVM to FreeBSD, instead of just porting KVM to
Illumos?

~~~
rednaught
FreeBSD has already had ZFS and DTrace for years. They have Jails which would
have benefited from Solaris' stronger Zones/Containers.

There are already a lot of BSD users. Where are the Illumos users? Are you
suggesting that all BSD users go to SmartOS? I used HP-UX and AIX for many
years...the one off UNIX systems just end up recreating each other due to
fragmentation. They are not the way forward. The smaller mind share/usage of
OpenSolaris/Illumos isn't helping. Wouldn't contribution to a BSD have
strengthened the community?

~~~
dap
Not at all. The beauty of KVM on SmartOS is that Joyent's customers can run
BSD, or Linux, or Windows, or whatever they want without even knowing it's
running on top of Illumos. That's why the user base isn't as much an issue.
But the underlying foundation is critical. I'm actually really glad BSD is
adopting DTrace and I'm sure it will be successful, but the BSD documentation
says that it's experimental and not yet production ready
([http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dt...](http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dtrace.html)).
Solaris DTrace has been _in production_ since at least 2005.

Disclaimer: I work for Joyent, but I do not speak for them.

~~~
moe
_without even knowing it's running on top of Illumos_

What's the advantage of running Linux->KVM->Illumos instead of
Linux->KVM->Linux?

~~~
acdha
DTrace and ZFS are both great and lack competitive alternatives on Linux. For
years, the horrid state of Solaris' userland made it hard to justify order-of-
magnitude support increases but this lets you use the best parts without
having to suffer through things like a system-destroying package updater[1]
simply to get a reliable filesystem.

1\. At a previous employer, we bought Sun compute nodes. Nice hardware,
arrived with Solaris 10 preinstalled. Each time we got a new shipment I'd get
them racked and see if the updater wouldn't render the system unbootable after
the first run. Second reboot was always to the Debian installer.

~~~
moe
Well, pretty much the same here. We ran linux on xfires until recently because
we loved the hardware, but wouldn't touch solaris with a 10ft pole.

I'm skeptical DTrace and ZFS can justify the risk to invest in a niche
platform. Personally I'm holding out for the linux alternatives (btrfs, ceph)
to mature.

------
darklajid
The biggest yay for me is in the fine print:

"SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level
virtualization and next-generation KVM) with __BSD package management and a
GNU toolchain__."

Oh god, this makes me happy. I gave Solaris a couple of tries, but I've never
felt at home there. These two facts mean that I'll try again.

~~~
binarycrusader
Solaris 11 Express has the GNU toolchain and a modern package management
system.

For those that want something like SmartOS but free, and with a modern package
management system, I would suggest OpenIndiana. OpenIndiana will be
integrating many of the SmartOS changes.

------
wmf
They're really going out of their way to not say "Solaris".

~~~
po
They're really going out of their way to not _use_ Solaris.

 _SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level
virtualization and next-generation KVM) with BSD package management and a GNU
toolchain._

I used Solaris for almost 5 years and when Sun was bought I thought for sure
the cool parts of it were doomed. Illumos popped up and I thought that they
didn't have a chance, it would wither and die. It's very nice to see that is
not the case.

~~~
wmf
_They're really going out of their way to not use Solaris._

Considering that the Illumos kernel is basically the OpenSolaris kernel, I
don't understand what you mean. They are using Solaris, but for some reason
they're trying to hide it.

~~~
po
This project is certainly not using _Solaris_ which is now an Oracle product
(and trademark) and not open source. It is based on Illumos (which originally
came from an _OpenSolaris_ kernel as you said) but my understanding is that
OpenSolaris as a project is dead. I don't think it's hiding something to name-
check Illumos instead of OpenSolaris.

------
mkup
Better they had ported KVM to FreeBSD.

Who will add support for the new hardware to basically abandoned OpenSolaris
kernel? Oracle? No. That small company? No. Leveraging FreeBSD community to
grow the list of supported hardware would be much, much wiser.

------
kraemate
This is simply amazing. Tracing KVM using Dtrace is going to throw up some
pretty useful and surprising results. And add ZFS on top that as well - with
all its features (sp. dedup+COW) useful for virtualized hosting.

Anyone know if linux has any variant/clone of dtrace yet? (and no, not
systemtap)

~~~
otoburb
Paul Fox seems to have been working on a Linux DTrace port since 2008 (based
on the directory listing). He blogs about his progress on DTrace (and other
things) at <http://crtags.blogspot.com/>

~~~
sogrady
Here's Bryan Cantrill, one of the original creators of DTrace, on the status
of the port:

[https://plus.google.com/105843697186982227624/posts/5uk5SHrh...](https://plus.google.com/105843697186982227624/posts/5uk5SHrhPt6)

------
nathanb
"ZFS makes SANs and other expensive, redundant storage systems obsolete"

Yes, because clearly what enterprise-level storage customers really want is to
roll their own storage solutions using open source technology.

~~~
daeken
Many, many companies do want this, yes. Back in 2005-2006, I worked at
MP3tunes and among other things I helped design the storage system. After
testing a bunch of storage solutions (many of them high-end systems), we ended
up rolling our own using Linux + MogileFS. This was scaled up to a couple
hundred terabytes before I left, and even higher afterwards. Sometimes the
off-the-shelf solutions simply don't work, especially at scale.

~~~
mitchty
Pretty much, the amount of times we've had issues with new storage vendor
arrays from a 3 letter company that starts with an E, and ends with a C, is a
bit too much to count.

Firmware bugs that affect anything on a fabric, that affect how it distributes
its cache slot locks on writes, issues with the entire array acting funny,
which are blamed on either the server hardware or os itself until proven to be
an issue with the array (this is far too common to be honest, yay for
"support" contracts), etc....

Yes you get "support", and I use the term lightly, with big vendors, but you
also have to take a machete through their support organization to get to
someone that can help you with a problem. At times rolling your own solution
will end up being both cheaper and less problematic. The old adage of "Nobody
ever got fired for choosing IBM" may soothe managers minds, but wait until you
do end up buying those fancy pants high end arrays and find out how much snake
oil turns out not to work on them.

~~~
acdha
We had great experiences with NetApp's products and support but it was priced
accordingly.

------
sciurus
Can anyone either explain or link to material that explains the relationships
between the different projects and distributions that have sprung up since
Oracle discontinued OpenSolaris?

~~~
piotrSikora
After OpenSolaris OS/Net kernel development was closed, 3 forks were made:
Illumos, SchilliX-ON and Stormix (sadly, they all seem quite inactive).

There are 3 distributions based on the first one: OpenIndiana, Nexenta and
SmartOS. There is also SchilliX (based on SchilliX-ON) and StromOS (based on
Stormix).

This is just based on my existing knowledge, so there might be more forks
and/or distributions out there.

~~~
binarycrusader
Both IllumOS and OpenIndiana are quite active (which shouldn't be surprising
since the latter relies on the former).

~~~
piotrSikora
Is it, really? According to GitHub's commit history, there were 297 commits
made since onnv_147, so ~1 commit/day. Prior to the fork, OpenSolaris was
getting ~10 commits/day.

To put this into perspective, there were over 8000 commits made to OpenBSD in
the same time-frame. I won't even try to compare this with FreeBSD or Linux...

Don't get me wrong, I love the effort, but at the same time I feel like there
wasn't any real progress made since the project's inception.

~~~
binarycrusader
It depends on your definition of active.

As an open source project, it's definitely active.

Is it as active as the actual Solaris codebase? No. But it has also
significantly less developers working on it.

~~~
piotrSikora
Well, it's active as in "not dead", but not as in "actively developed
operating system".

I'm aware of the difference in the number of developers, but you have to agree
that there are hardly any user-visible changes since onnv_147.

~~~
binarycrusader
Uh, no, it is definitely an "actively developed operating system".

As for user visible, actually, yes, changes made to dtrace recently for
example.

Sparse zones as another example, and so on.

Remember that Solaris (and its derivatives, unlike Linux) is userland + kernel
-- not just the kernel. So changes in userland count in my opinion when
considering "actively developed operating system".

~~~
piotrSikora
Apparently our definitions vary a bit ;)

> As for user visible, actually, yes, changes made to dtrace recently for
> example.

> Sparse zones as another example, and so on.

Those changes came from SmartOS/Joyent and were not made by the IllumOS
developers.

Don't get me wrong, I really appreciate their work, but I've already lost hope
with IllumOS/OpenIndiana... Fortunately Joyent stepped in, so I hope that
things will speed-up a bit now :)

------
jjm
Cool. I see this really helping me when testing my application prior to
deployment. Would like to see some more movement with No.de. Have been waiting
for a bit to get access.

------
jkahn
I'm struggling to understand why ZFS makes enterprise storage obsolete. One of
the benefits of the SAN is that it is shared amongst a number of servers.

How does ZFS help with that? Assuming you install SmartOS as virtualisation
host, you'd still need some kind of shared storage.

[Edit:] The reason I point this out is that if you lose a host, you lose all
VMs running on that host. And then you won't easily be able to get them back
online on another host if you've lost your storage as well.

~~~
wmf
You can use ZFS+Comstar to create a SAN controller.

~~~
organico
But you were able to do this before SmartOS... I'm curious about this too, and
how Joyent have architected their cloud arrangement. I feel like perhaps
hypervisors have local ZFS storage, as opposed to large SAN-backed virtual
machines, but I can't really find much useful information on how
Joyent/SmartOS implement/architect storage...

------
kermitthehermit
This looks very good.

How can packages be added? How can it be reconfigured? How is one supposed to
add management tools (think libvirt, openstack) to it?

I took it for a spin and noticed it had kvm, a whole lot of regular Illumos
binaries and that it had a BSD-ish userland.

I couldn't find anything like libvirtd, I didn't get to check if it has it,
though.

------
sgt
Fantastic news. I've been excited about Illumos for a long time, and now we're
finally getting a decent distro with commercial backing. My servers will be
very happy when they hear about this.

------
RexRollman
As an OS geek, I always enjoy the annoucement of a new OS, even if it is one I
personally have no use for. Choice is good.

------
foobarbazoo
Would be sweet to run SmartOS on Backblaze's storage hardware.

------
ristretto
I 'm happy for all the involvement of Joyent in opensource, but how about also
dropping the prices in their accelerators? I 've been using the same servers
with the same prices for 3 years with zero upgrades.

