
Triton: Docker and the “best of all worlds” - thepumpkin1979
https://www.joyent.com/blog/triton-docker-and-the-best-of-all-worlds
======
themgt
I find these to be pretty weasel words: _Is Triton open source?

Yes. Triton Elastic Container Infrastructure is the commercial offering built
on the top of the open-source SmartDataCenter cloud infrastructure management
platform. In addition to the open source components, Triton includes support,
a DevOps portal, and intellectual property indemnification only available to
commercial customers._

In more plain English, "Yes. Triton is open source just like Mac OS X"

[https://www.joyent.com/developers/triton-
faq#os](https://www.joyent.com/developers/triton-faq#os)

~~~
lloydde
That's a good thing to clarify. It's not an open core system like Mac OS X.
SmartDataCenter is open source and Triton is our product branding.
[https://github.com/joyent/sdc](https://github.com/joyent/sdc)

There is work in progress to clean up and open source the self-serve portal
and its dependencies.

The propriety bits are not ours: "The Joyent SDC USB images offered to our
customers contain certain firmware binaries for use with Joyent-branded
hardware. We cannot make that software generally available in the open source
builds; however, it is available to all who have purchased Joyent-branded
hardware and is of no use without that hardware."
[https://docs.joyent.com/sdc7/obtaining-
software](https://docs.joyent.com/sdc7/obtaining-software)

~~~
themgt
But what you're branding Triton in fact appears to be a PaaS GUI on top of
SDC? i.e. the way people manage their containers/hw in Triton the service, is
not available in SDC which seems to be all CLI tools?

~~~
lloydde
As the Docker Engine for SDC is being developed to the Docker API and CLI,
Triton can be used through any Docker ux. [https://github.com/joyent/sdc-
docker/blob/master/docs/diverg...](https://github.com/joyent/sdc-
docker/blob/master/docs/divergence.md)

The interesting bits are not the self-serve portal. The portal is good for
introducing the stack and supplementing the experience, but few use a portal
for infrastructure management -- they require more automation, specific to
their business.

We're working hard on getting the self-server portal code architecture and
hygiene up to open sourcing it. From there we can get these portal plugins
open as well. Our on-prem customers are hungry for this, as many of them have
a mandate for OSS.

------
thinkingkong
Tldr.

Triton lets you run your docker containers in the cloud without worrying about
virtual machines or setting up your own paas.

Signup here:
[https://www.joyent.com/lp/preview](https://www.joyent.com/lp/preview)

~~~
hueving
So how is this different from the container service from google?

~~~
brianshaler
One interesting bit from the article: "Does this mean that I'll be able to
DTrace my Linux apps in a Docker container?" (yes)

Technically you can run dtrace on various platforms, and I don't know enough
to tell you specific reasons why you would rather use dtrace on smartos than
linux. I got the impression a few years ago the ports weren't as good.
(perhaps relating to the linux port not yet being a kernel module?)

~~~
lloydde
Debugging LX branded zones on SmartOS

[https://www.youtube.com/watch?v=6oIBiWdh41c](https://www.youtube.com/watch?v=6oIBiWdh41c)

"A gritty how-to on debugging LX branded zones on SmartOS presented internally
at Joyent on 1/30/15\. While given to an internal audience, there is quite a
bit of information here that may be interesting to the broader community"

------
DannoHung
Uh, so I've never really heard of SmartOS or Joyent's SDC before now...

This stuff looks pretty fucking unreal. What's the catch? It looks like it
solves so many problems with clustered container deployment. Exposing a whole
datacenter as a single Docker host seems like the end-game to me.

Am I missing something? I'm a little low on sleep today.

edit: To answer my own question a little: I guess the docker-sdc isn't quite
fully baked yet. That's not really a huge issue since it's still a preview, I
think?

~~~
bananaoomarang
Nope, SmartOS is pretty neat.

~~~
ssmoot
Ditto. We were early adopters at work. KVM + ZFS and practically no setup is
ace.

------
SandB0x
Can someone explain this in words that a time traveller from 2010 would
understand?

~~~
mrfusion
Love the question. This should be the new ELI5. I think a lot of articles on
here would benefit from answering that question.

~~~
raus22
The Explain it like i have been in a coma for 5 years... Elihbiacf5y or just 5
years in coma, explain! 5YICE....

Or ETTT5YA or ET5... explain to time traveler from 5 years ago

------
josh2600
Joyent has solved some really hard problems to make this work. I was lucky
enough to get the chance to sit down with Bryan last week and talk with him
about what this took. It's essentially reimplementing Linux on top of
Illumnos, which is no tiny feat.

There are still messy hurdles to running Docker in production, but it's clear
that Joyent has really tried to make something awesome here.

As someone who works deeply in the future-container space, I applaud folks who
are taking us deeper down the rabbit hole. I think it's clear that this whole
'back-to-the-future' isolation technology is really cool stuff.
Jails/zones/containers have been around forever and I think it's great that
we're finally taking advantage of this technology.

Edit: at Terminal.com, we want people to push Linux forward, and this is a
great example of taking Linux to new and intriguing heights. I did not think
we would have Linux on Illumnos in quite this way in 2015 and it's delightful
to see. We are all standing on the shoulders of giants and it's great to reach
new vistas.

~~~
justincormack
Isn't it an example of taking Solaris to new and intriguing heights? There is
no actual Linux code in the implementation.

~~~
josh2600
That's a great point, but I think the point of what Bryan has been doing is to
make Linux work with Zones (and dtrace).

That's a primitive that Joyent has wanted to upstream into the Linux kernel
for a long time and has never been able to get the necessary consensus around
it (similar to OpenVZ's troubles getting their work upstreamed).

In short, this is sort of a hack to give you zones on Linux without needing to
get zones into the upstream. Yes, there's no linux code, but there is a lot of
required understanding of Linux code to make something like this work.

It's kinda amazing that they got 64-bit linux to run on top of Illumnos,
right? I did not see that coming and maybe that's because I'm ignorant in some
capacity, but it's been a pleasant surprise.

~~~
justincormack
Emulations have been a Unix feature for a long time actually. NetBSD has had
64 bit Linux emulation for ages, for example, but its not very complete
because no one has cared enough to implement more. For example Illumos is
AFAIK the first system to emulate epoll. The Linux API is huge and
historically the process has been just fixing stuff for a binary someone wants
to run. It is very tedious work...

I dont really see it as zones in linux. More a gateway drug for non-Linux.

~~~
bch
Hey Justin -- do you have insight into how hard any _particular_ remapping
(ie: epoll) is to perform ? I was talking to @bcantrill about their effort at
a Docker meetup and mentioned the NetBSD emulation (he said "Oh! Of course!"),
but whats interesting (in retrospect) is that they (Joyent) just tried running
stuff and played whack-a-mole w/ unimplemented APIs... how tough would it be
for "us" (NetBSD) to occasionally implement pieces ?

edit: parens

~~~
bcantrill
Speaking without familiarity with NetBSD, I think it depends on what kernel
facilities the system happens to have; speaking for SmartOS/illumos, in many
cases we were able to slightly rephrase Linux facilities as extant facilities
-- saving a considerable amount of time and effort. For example, the big
realization with epoll was just how naive it is -- so much so, in fact, that
it actually looks very similar to a pre-port mechanism (/dev/poll) that we
developed nearly 20 years ago (!!) and later deprecated in favor of ports.
epoll would have been much nastier without /dev/poll -- which is likely the
greatest service that /dev/poll has ever provided anyone...

~~~
bch
Hi Bryan -- I'm also aware that epoll may have been a bad example on my part,
because isn't it subject to some nasty fork/share bugs wrt handling the (well)
handle, and what file it's _actually_ associated with the handle -- so a
parent can get notifications on a handle it doesn't have, or worse,
notifications for a socket that it _does_ have that is not _really_ the same
handle that's issuing the event.

In cases like that, did you end up trying to be bug-compatible, or make a
design decision to clear up the trouble ?

[edit -- spell "Bryan" correctly]

~~~
bcantrill
Funny you should mention that one in particular -- from our (SmartOS's)
epoll(5) man page:

    
    
           While  a  best effort has been made to mimic the Linux semantics, there
           are some semantics that are too  peculiar  or  ill-conceived  to  merit
           accommodation.   In  particular,  the  Linux  epoll facility will -- by
           design -- continue to  generate  events  for  closed  file  descriptors
           where/when  the underlying file description remains open.  For example,
           if one were to fork(2) and subsequently close an actively epoll'd  file
           descriptor  in  the  parent,  any  events generated in the child on the
           implicitly duplicated file descriptor will continue to be delivered  to
           the parent -- despite the fact that the parent itself no longer has any
           notion of the file description!  This epoll facility refuses  to  honor
           these  semantics;  closing  the  EPOLL_CTL_ADD'd  file  descriptor will
           always result in no further  events  being  generated  for  that  event
           description.
    

So while we do aspire to be bug-compatible, we're not about to compromise our
principles over it. More details (or some of them, anyway) can be found in the
talk on LX-branded zones that I gave at illumos Day at Surge 2014.[1][2]

[1] [http://www.slideshare.net/bcantrill/illumos-
lx](http://www.slideshare.net/bcantrill/illumos-lx)

[2]
[https://www.youtube.com/watch?v=TrfD3pC0VSs](https://www.youtube.com/watch?v=TrfD3pC0VSs)

------
kordless
Here's the signup link for Joyent's hosted container service:
[https://www.joyent.com/lp/preview](https://www.joyent.com/lp/preview).

We're seeing the initial explosion of the microservices ecosystem right now.
I've been spending most of my time over the last few years thinking about
trusted decentralized infrastructure and have decided that microservices,
including hosted ones, could be one possible solution for instantiating trust
as a proper 'knob' of the cloud. This Intracloud, if I'm allowed to use a new
buzz term, will be smeared across 100s (1000s?) of datacenters world-wide.
Have a high-trust use-case? Run it in a German or Dutch datacenter. Have a
high-efficiency use case? Run it on a friend's cluster for free.

I believe in this idea of the 'trust knob' enough that I sought out a job with
the folks at [https://giantswarm.io/](https://giantswarm.io/), where I am now
a dev evangelist. We are a German hosted microservices stack which provides a
Docker platform. Alpha signup is here: [https://giantswarm.io/request-
invite/](https://giantswarm.io/request-invite/). Demo of it in action here:
[https://github.com/kordless/swarm-ngrok#ngrokn-giant-
swarm](https://github.com/kordless/swarm-ngrok#ngrokn-giant-swarm)

I also hacked together a SF Microservices meetup last Friday:
[http://www.meetup.com/SF-Microservices/](http://www.meetup.com/SF-
Microservices/). 139 people have already joined. Would like some feedback for
content! Planning on mid-April for the first event.

I'm excited to see what happens with this market in the next 6 months!

------
batbomb

        > When Docker first rocketed
    

I thought this was an interesting way to start off the post.

------
phildougherty
"You can run your Docker containers across entire data centers without ever
creating a "cluster" as other IaaS providers would have you do."

This is a little unclear. I get that you don't have to create a "cluster"
because you're going to use the API to launch containers on existing dedicated
hardware, but how is that a WIN for your finance team? I guess I'm missing
something.. but wouldn't you have to already have systems online that you're
paying for that these containers can be launched on?

From: [https://www.joyent.com/blog/docker-bake-off-aws-vs-
joyent](https://www.joyent.com/blog/docker-bake-off-aws-vs-joyent)

~~~
thinkingkong
The difference for the operator is: Baremetal -> smartos -> container vs
Baremetal -> hypervisor -> virtual machine -> container.

Theres root safety in the smartos implementation of docker. So they can do
multi-tenancy.

For the customer, you dont have to provision entire virtual machines to run
docker containers.

~~~
phildougherty
Is it multi-tenancy of various Joyent customers on the same hardware? So you
get containers placed somewhere in their data center on shared hardware with
other customers? Or do you already need to have dedicated hardware provisioned
at Joyent to launch containers on? I'm still unclear on that point, which
makes me question how it's saving the end user money.

~~~
xj9
Yes, you are running on shared hardware, securely, but with no hypervisor.

The biggest money-saver is in performance. IO-heavy applications can see
anywhere from 5-10x performance improvements[1] by switching from hardware- to
OS-virtualization.

[1]: [http://dtrace.org/blogs/brendan/2013/01/11/virtualization-
pe...](http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-
zones-kvm-xen/)

------
yonran
Very cool. Is the next step integration with Manta so you can specify an
environment of Linux executables to map over your data?

~~~
dap
Yes, that's definitely the direction that we're headed.

Manta allows users to spin up large numbers of transient containers around
their data (i.e., without moving data around) in order to do map-reduce
operations, and without having to think about the container management. Today,
these compute tasks are specified as scripts that run inside a well-stocked
SmartOS container, plus some optional assets to download (for custom binaries
or other data). The obvious next step would be to let people specify they want
to run an LX zone (rather than a SmartOS zone) so that they can run GNU/Linux
binaries. Then we can consider whether it makes sense to phrase the task as a
Docker image. That might well be a more natural way to incorporate a larger
bunch of binaries and other shared data. Architecturally, all of this should
be relatively straightforward (famous last words!).

------
themgt
Didn't see the actual repo linked: [https://github.com/joyent/sdc-
docker](https://github.com/joyent/sdc-docker)

"A Docker Engine for SmartDataCenter, where the whole DC is exposed as a
single docker host. The Docker remote API is served from a 'docker' core SDC
zone (built from this repo). ... Disclaimer: This is still very much alpha.
Use at your own risk!"

~~~
ochoseis
To me it seems like Triton is about shoehorning Linux into Solaris/SmartOS
containers. While this seems great for accelerating adoption, why not just use
SmartOS-specific Dockerfiles/repos in conjunction with sdc-docker (Not
directly addressing the parent, more of a general question)? Imagine all of
the crazy bugs and edge cases that will arise when trying to run Linux
binaries on UNIX.

I'm surely making some assumptions, but the primary languages/developers that
something like this targets are probably like Java, Javascript/Node.js, Ruby,
Python, C...which are probably all portable enough to run on SmartOS without
much/any modification.

~~~
wmf
Docker multiarch support isn't finished yet, and getting all the world's
Dockerfiles rebuilt on SmartOS is a big task.

There are also different philosophies about whether container images should be
distributed as source (Dockerfiles) or as binaries. It looks like Joyent
decided to run Linux/x86-64 binaries to be on the safe side.

------
mrinterweb
Is there any pricing information for Triton yet?

~~~
jholmes83
Pricing here: [https://www.joyent.com/blog/expanded-container-service-
previ...](https://www.joyent.com/blog/expanded-container-service-preview)

~~~
mrinterweb
Thanks for finding that link. That is a very attractive pricing model. I am
currently running about 10 VPS running about 25 containers. The costs of the
VPS hosting is affordable, but managing those VPS hosts, deployment,
balancing, discovery, and scaling are the hard parts. I'm looking forward to
learning more about Triton.

------
kstenerud
Is there any further work going on to support AMD in the KVM driver? I
remember the codebase having sat idle for a few years last time I checked...

~~~
snw
While sadly not merged upstream yet, it works very well. I've been running
custom SmartOS "eait" builds from
[http://imgapi.uqcloud.net/builds](http://imgapi.uqcloud.net/builds) (which
include AMD support) for some time now without any issues. The source can be
found at [https://github.com/arekinath/smartos-
live](https://github.com/arekinath/smartos-live)

------
Bedon292
This looks like a very nice way to deploy containers. I see the per hour
prices ([https://www.joyent.com/blog/expanded-container-service-
previ...](https://www.joyent.com/blog/expanded-container-service-preview)) but
is there any pricing on bandwidth? I imagine there will be eventually, right?

~~~
adamtaro
I'd suggest looking at this page as an indicator
([https://www.joyent.com/public-cloud/pricing](https://www.joyent.com/public-
cloud/pricing) \-- Bandwidth tab): all inbound traffic free, outbound is 12
cents/GB after the first GB, cheaper at higher tiers.

------
crashoverdrive
I feel like perhaps I'm the only one concerned about this: (Somehow, “SmartOS
+ LX-branded zones + SmartDataCenter + sdc-portolan + sdc-docker” was a bit of
a mouthful.)

Not only is it a mouthful, it means there is a much wider space for things to
go wrong.

~~~
tracker1
Compared to say "hypervisor + Linux + kubernetes + weave + etcd + lxc-docker"
...? Not to mention all the pieces that work together for lxc-docker.

------
23david
Excited to see this. Congrats to the team on the release!

------
jimpick
Nice!

------
bau5
I saw the name and my desire to read it plummeted. The same person wrote
[http://www.joyent.com/blog/the-power-of-a-
pronoun](http://www.joyent.com/blog/the-power-of-a-pronoun) which led to one
of the most talented developers of node core and libuv leaving the project for
a while.

~~~
nosequel
That "most talented developers of node core and libuv" took time out of his
day to revert a commit to take a gender neutral pronoun and make it back into
a male one. Talented people can be shitbirds also you know.

------
redwood631
This is running Linux binaries on Oracle Solaris (which they call SmartOS),
and not the real thing that Oracle is still developing, but a fork from 2010
that is maintained by 1/10th of the developers it once had.
[https://www.openhub.net/p/illumos](https://www.openhub.net/p/illumos).

This would have been pretty rad 10 years ago, when the world still cared about
Solaris.

On a minor note: the post doesn't credit Oracle or Solaris, from where more
than 90% of their SmartOS code comes from, until Oracle closed their code in
2010.

~~~
CamTin
This is cool RIGHT NOW because WHO THE HELL CARES WHAT THE UNDERLYING OS IS?
For a user trying to bring something to market, here's what matters:

1) it's cheaper (theoretically, due to performance savings from not dealing
with hardware virtualization), and 2) it looks like a docker host

I honestly expected Docker Hub to expand into this kind of operation before
anyone else, but Joyent, from out of nowhere, has done it instead. Why should
anybody care that they're doing it with a Solaris-based kernel, Linux, on
NetBSD, as MAME machines, VIC-20 cartridges, or with 1402 machine images and
druidic spells?

To the user, it's just a magical world where your Docker host always expands
to whatever size you need it to (and you of course get a bill for the usage).

The real test will be whether or not this is cost-competitive to deploying
Docker swarms on other public cloud providers. If the performance and tenancy
claims are as they claim (or even the same order of magnitude, which seems
reasonable), and they're not insanely greedy with pricing, this could be the
biggest tech since Linux itself.

~~~
jgrowl
I was wondering how this would compete with docker swarm. Is my understanding
right that docker swarm basically lets you run an agent on individual docker
hosts whereas triton will let you run a single virtual docker host?

~~~
CamTin
Docker swarm basically lets you say "I want to deploy this image somewhere,
you choose from the available actual machines" based on constraints that
you've set up. You also had to actually set up the machines themselves (though
docker-machine is intended to take some of the pain out of this).

sdc-docker (the tech behind this, Triton) basically lets you just keep
deploying containers without any regard for the actual underlying machines (or
EC2 instances, DO droplets, Linodes) that you are actually deploying to. It's
like docker-machine + docker-swarm but all automatic and managed by guys on
the other side of the curtain, so you can just write the checks and keep the
Docker goodness flowing.

~~~
nemothekid
> _sdc-docker (the tech behind this, Triton) basically lets you just keep
> deploying containers without any regard for the actual underlying machines
> (or EC2 instances, DO droplets, Linodes) that you are actually deploying to.
> It 's like docker-machine + docker-swarm but all automatic and managed by
> guys on the other side of the curtain, so you can just write the checks and
> keep the Docker goodness flowing._

Not sure I get this - Is this any different from mesos/kubernetes? If I am
running this software and deploying my own datacenter _someone_ stills need to
provision the machines.

~~~
jgrowl
I'm still fuzzy on this also, but it sounded like it would fill a very similar
role as kubernetes except that they are claiming superior performance.

~~~
lloydde
You've got it. This is a complete stack (sdc + lx-brand, sdc-docker,
vxlan/portolan) fully benefiting from Illumos's architected OS virtualization.

