
Runit – a Unix init scheme with service supervision - jsingleton
http://smarden.org/runit/
======
atom_enger
Runit is amazing. I've used it on several large scale websites with great
success. Runit follows the unix philosophy of being stupid simple and doing
one thing incredibly well. If you're starting up a new project, consider using
Runit.

~~~
paulsmith
Not only is it simple and does the do-the-one-thing-well thing well, which is
true, it's also _correct_. It's hard to overstate how valuable runit is in
production because of that. It does the right thing with regard to clearing
the environment, detaching from controlling terminal, logging, and many other
subtle aspects of operating a service. I never worry about runit.

~~~
freshhawk
Back when I changed from supervisord to runit my life was substantially
improved. That correctness means way fewer emergency maintenance ops issues in
production.

~~~
general_failure
What is wrong with supervisord?

~~~
freshhawk
First of all and mostly, we could never exactly figure it out what was going
wrong.

This was a few years ago and I don't remember all the details. We just had
occasional issues with stopping/starting and especially restarting processes
when pushing a new version out or when a process crashed. I do remember it
could occasionally report a successful restart and still leave the old
process(es) running.

All my developers became very familiar with supervisord, and it was number 2
on the troubleshooting list (1. Did we introduce a bug in a recent commit? 2.
Did supervisord do something weird again).

After we switched to runit, only devs that touched ops knew about it at all.
And we all forgot it was there. That's what you want in an ops tool.

------
davexunit
I use GNU dmd instead. Simple and very extensible with Scheme. I use it as PID
1 on 2 of my machines, but I use another instance of it as a user service
manager on all of my machines. I've also been meaning to replace runit with
dmd in Phusion's passenger-docker image to get something more hackable.

[https://gnu.org/s/dmd](https://gnu.org/s/dmd)

~~~
kragen
Having dynamic memory allocation, let alone garbage collection, in my pid 1
doesn't sounds like a great idea.

(I know, as long as I'm using Linux, I'm stuck with dynamic memory allocation
_in the fucking kernel_. I don't have a plan for how to fix that yet.)

(Also, thank you so much for all your help with Guix this last week!)

~~~
blackbeard
There's nothing wrong with dynamic allocation in a kernel. It is for example
better than having fixed size process tables and all the crap that comes with
that.

"holy crap I've got to recompile my kernel to get more processes" is so
1995...

~~~
kragen
You know what else is "so 1995"? 400-day uptimes.

~~~
blackbeard
Not really. I have centos boxes that have been up for over 3 years.

------
nisa
If you run runit you can also take a look at runwhen:
[http://code.dogmap.org/runwhen/](http://code.dogmap.org/runwhen/)

It's cron and at implemented in a quite elegant way.

Unfortunately there does not seem to be much love for it in distributions.

[http://code.dogmap.org/runwhen/overview/#rationale](http://code.dogmap.org/runwhen/overview/#rationale)

------
edwintorok
There is a good comparison/documentation of what an init system like runit
should do and why on the S6 site:
[http://skarnet.org/software/s6/why.html](http://skarnet.org/software/s6/why.html)
[http://skarnet.org/software/s6/overview.html](http://skarnet.org/software/s6/overview.html)

Runit has the advantage that it is packaged in Debian and you can start using
it right away.

~~~
dkubb
Do you have any experience with S6? Do you know how it compares to runit?

~~~
cathexis
I've used s6 a lot, though not as an init replacement in any systems that
matter. As a supervisor I think it's great. For basic stuff (root process
supervising supervisors supervising daemons) the two are basically identical
with some superficial differences. For wider ranging stuff s6 is really good
since it has a bunch of ancillary programs that solve a lot of serious issues
in full system supervision (readyness notification vs polling in a run script,
ucspi socket handing, etc).

~~~
cathexis
I missed actually answering the question about the difference between the two
(plus, that comment is in dire need of editing and the edit window appears to
have expired).

The superficial differences are things like: service to logger pipe holding
happens at different levels of the supervision tree (the root `s6-svscan'
holds them in s6, the per-service runsv holds it in runit), `s6-svc -CMD
behavior cannot currently be overridden whereas you can with `sv CMD',
`s6-svscan' will immediately re-scan its directory with SIGALRM whereas
`runsvdir' only polls for changes on a 5 second timer.

For basic supervision tasksboth are great, with runit being the simpler of the
two in terms of understanding what it gives out of the box. For larger tasks
(full system supervision, inter-service ordering dependencies, etc) s6 has the
tools to make that easy whereas with runit you're going to find yourself
playing stupid tricks in run scripts to get similar behavior.

------
pchm
A genuine question, as it's not quite clear to me: can someone explain like
I'm 5, what can I use runit for exactly? Is it a replacement for init.d or
something more like monit? I always use monit to keep my Rails-related
processes running and I've read somewhere that using monit combined with runit
may be a better solution, but isn't it a bit redundant to have the two at the
same time?

~~~
cathexis
Essentially it's an alternative method for starting long running services (and
restarting them if they fail).

The things all daemontools-inspired process supervisors do: Runs a scanner
gainst a service directory containing a directory for each program you want
supervised (classically these are symlinks to another part of the system). The
scanner spawns a supervisor program for every directory it finds which then
looks for a program called `run' inside that directory and runs it as a
foreground child. If the supervisor's child stops, it runs an optional script
in it's supervision directory called `finish', and then runs run again. If one
of the scanner's child supervisors stops, the scanner spawns it again.

In runit's case, the scanner is called `runsvdir', the supervisors are called
`runsv', and it also comes with a program called `runit' that can act as a
replacement for your PID1 init whose sole job outside of boot and shutdown
time is to resurrect runsvdir if it exits. Note that runsvdir is perfectly
happy to be run via an entry in your /etc/inittab if you're running under
sysvinit.

At their most simplistic, that's all process supervision is - a bomb-proof way
of keeping services running. All supervisors in the daemontools family come
with a control program to interact with the process supervisor, as well as a
stdin-based logger that doesn't rely on syslog. The main benefits it brings
over the classic init.d/ model are simplicity (the most complex run script I
have is 14 lines of dead simple shell, most are 4-5) and automatic
restartability which init.d/ daemonization doesn't have.

From what I understand from monit's manpage, it's a full-bore rule-basd system
monitor. Generally speaking, process supervision is tacking one problem (keep
daemons running), whereas monit is tacking another (system state monitoring).
Yes monit can act as a process supervisor, but it does so by polling the
system state and hooking into the existing daemonization infrastructure.

------
manas952
Runit is fantastic. If you are using Chef - the runit cookbook integrates very
nicely.
[https://supermarket.chef.io/cookbooks/runit](https://supermarket.chef.io/cookbooks/runit)

~~~
atom_enger
Now that I think of it, the chef cookbook for Runit is what made it so easy to
deploy to production. The 'runit_service' resource was absolutely invaluable.
Forget the complicated upstart stanzas or dealing with supervisord, just write
a shell script to run your program in the proper environment and ba! You've
got a service!

------
jlongster
Nice to see daemontools-inspired work on HN in the past few days. I've been
using runit for years, and as other comments have said, it's just not
something I ever have to worry about. When I look around at other solutions it
seems like I'd be taking a huge step back (making things more complex with
arguably less usefulness).

I've been meaning to do some blog posts about runit, as it seems like it sits
on the back-burner in general. Does anyone know how actively it's maintained,
or has it just reached such stability that it doesn't need much maintenance?
It would neat to see it on github with some real docs and such.

------
idop
And just a friendly reminder that I wrote a shell for runit (and other
daemontools-based supervisors) called svsh at
[https://ido50.github.io/Svsh](https://ido50.github.io/Svsh).

------
lugus35
Voidlinux uses runit by default.
[http://www.voidlinux.eu](http://www.voidlinux.eu)

------
jedisct1
runit is what the Phusion Docker base image
[http://phusion.github.io/baseimage-
docker/](http://phusion.github.io/baseimage-docker/) is using, and it's the
perfect tool to start and supervise containerized apps. I also love the fact
that it can wait for a service to run (in order to wait for dependencies), and
that stuck services can be restarted in a more radical way than a single TERM
signal.

~~~
explorer666
I'm using this too (after suffering all the other supervisors), and I have no
clue why runit isn't the standard tool in Linux. Can any Linux expert explain
that?

~~~
vezzy-fnord
Linux is a kernel. It has no standards for userspace, though some have been
attempted for GNU/Linux in particular and there are _de facto_ conventions,
but that's it.

------
stephen-mw
Runit is the default system daemon for Phusion's baseimage-docker Ubuntu
image[0]. I've used it in a basic manner and have been very happy with it.

For containers it hits a real sweet spot: lightweight and easy to use within a
limited scope of processes.

[0] [https://github.com/phusion/baseimage-
docker](https://github.com/phusion/baseimage-docker)

------
clebio
Given the other glowing comments, maybe this would be a place to ask: should I
bother with Upstart, or Systemd? I see Shuttleworth announced the move to
systemd, but it's not available on Ubuntu 14.04 servers right now. I'm writing
provisioning for our production fleet, what should I use?

The vagaries of the OS wars make something like Runit tempting.

~~~
JoshTriplett
Depends heavily on what you need to provision, and whether you can select your
provisioned environment based on what you want to support. Upstart is, at this
point, effectively dead for any future distribution. However, if you want to
support existing Ubuntu LTS distributions for the remainder of their lifetime,
you'll need to handle it. Similarly, if you want to support the current rounds
of enterprise distributions, you still need to support sysvinit. And if you
have any non-Linux systems, you'll need to handle whatever they use as well.

On the other hand, if you can ensure that your production systems all run
relatively recent Linux distributions, you can safely assume systemd. Which
specific distributions you have in production will determine the oldest
version of systemd you have to support; since new features get added
regularly, you'll want to know the oldest version you can assume.

What are you trying to provision, and what requirements do you have?

~~~
clebio
I appreciate the detailed reply, and the others as well. To address your
questions and some of the others:

We can generally pin our entire fleet against whatever we want -- until we
can't. One particular (very large, very important) dependency currently
requires CentOS 6.5. Go figure.

As much as possible, I want to treat infrastructure as a commodity, so
catering to specific versions (even LTR versions) breaks that goal. As another
reply notes, if we lock in at Ubuntu 12.04 and it's predictable, that's
excellent.

As another mentions, Upstart was easy to get a handle of quickly. As an aside,
it bugs me that the (otherwise excellent and detailed) Upstart docs don't seem
to mention that all of Upstart is now deprecated. Hence the original
confusion! In general I _don 't_ want flexibility (not first and foremost), I
want simple and consistent.

~~~
JoshTriplett
> One particular (very large, very important) dependency currently requires
> CentOS 6.5.

What dependency is that? And can you run it in a single-purpose virtual
machine, rather than directly on real hardware, to make it easier to manage?

> As much as possible, I want to treat infrastructure as a commodity, so
> catering to specific versions (even LTR versions) breaks that goal. As
> another reply notes, if we lock in at Ubuntu 12.04 and it's predictable,
> that's excellent.

You definitely don't want to cater to specific versions any more than you have
to; ideally you want as few versions across your entire fleet as you can. If
you could get it down to just CentOS and the latest version of some up-to-date
distribution, that would help; if you could make CentOS a virtual machine
under that same up-to-date distribution, that's even better, insofar as you
can then manage all the physical machines identically.

> As another mentions, Upstart was easy to get a handle of quickly. As an
> aside, it bugs me that the (otherwise excellent and detailed) Upstart docs
> don't seem to mention that all of Upstart is now deprecated. Hence the
> original confusion!

The Upstart docs aren't going to say that Upstart is deprecated. It's more a
view of the Linux distribution ecosystem as a whole: the one distribution that
primarily drove upstart usage and adoption is switching to systemd, so there
won't be further momentum or resources behind upstart in the future. That
doesn't make it instantly obsolete, but it does mean that starting a new
project today around upstart is a bad idea.

(In fairness, there's one other notable distribution using upstart as well,
namely Chrome OS; however, that's a bit of a special case in several ways, and
in any case I hope to change that in the future.)

~~~
clebio
I'm wary of naming the dependency, because it's a vendor's solution and I'm
not trying to call them out, but it's their ecosystem of tools and it has to
run as the host OS, not virtualized. But all your comments are spot-on, and I
appreciate it.

The Upstart thing is just somewhat odd, because it's a Canonical project, and
Ubuntu is also Canonical's. At any rate, you're right that it's not
immediately lost. But trying to pin against even the long-term releases of
Ubuntu seems hopeless. I've been working on upgrading a 12.04 deployment to
14.04. There aren't any features that we're missing in the OS, so it's really
only a concern from a security or support perspective. The shop I'm in was
already using Ubuntu when I came on, but maybe the lesson here is to migrate
to Debian, rather than bother upgrading to the newer Ubuntu release.
(Migrating to CentOS would be somewhat more work.)

~~~
JoshTriplett
> The Upstart thing is just somewhat odd, because it's a Canonical project,
> and Ubuntu is also Canonical's.

That's why they pushed so hard for Debian to use Upstart. Once Debian switched
to systemd instead, Canonical announced that they would too.

> The shop I'm in was already using Ubuntu when I came on, but maybe the
> lesson here is to migrate to Debian, rather than bother upgrading to the
> newer Ubuntu release.

I'd certainly recommend that myself. Unless you have a hard requirement for an
"Enterprise" distribution (RHEL or SLES), I tend to advocate Debian stable
rather than Ubuntu LTS, _especially_ on a server.

------
kragen
I use runit. It's great. (Daemontools didn't have the ability to sleep for 30
seconds after my daemons crashed at startup because, like, some filesystem
wasn't mounted or something.)

~~~
Spiritus
How did you manage that? We currently have a problem with crashing services
using up all system resources trying to constantly restart. Or better yet,
exponential backoff.

~~~
ploxiln
In the recent past I worked at a place that used daemontools extensively. We
used a wrapper shell script which would do the restart loop detection, then
exec the actual process to run. If it detected X restarts in Y seconds (by
echoing the unix timestamp to a restart log and checking it) it would "svc -d
$(dirname $0)" (or something like that) and exit.

We also had a monitoring service (nagios) that would check if any services
"auto-downed" on any servers and alert us.

------
jflatow
Runit is / was awesome but how is this news?

~~~
bitwize
Many people still think that it's a 3-way horse race between sysvinit,
Upstart, and systemd (which systemd has all but won). The Debian technical
committee certainly acted like that was the case last year.

~~~
vezzy-fnord
A lot of people further neglect all the prior art in init replacements before
those. The simpleinit dependency mechanism, depinit, initng, minit, daemond,
Seth Nickell's GNOME experiments, eINIT, cinit and so forth.

Instead, the way it was presented in public is that the Linux distros had been
battling with brittle sysvinit scripts (true, but also largely self-inflicted)
for so long until systemd came in to heroically save the day.

It was pretty shocking to watch how the scene evolved from apathy to having an
urgent problem that must be solved now.

~~~
icebraining
What apathy? Ubuntu and Fedora had already switched to Upstart, and Gentoo to
OpenRC, due to the problems identified with sysvinit.

~~~
vezzy-fnord
OpenRC isn't an init daemon. It's only a process management framework, hence
the name. It's usually used in conjunction with sysvinit as PID1. OpenRC was
originally motivated by replacing the older baselayout scripts, from what I
recall.

Upstart didn't come about until later from many of the alternatives I listed,
and its origins were mostly in response to launchd. It was quite rudimentary
initially. [1]

[1]
[https://wiki.ubuntu.com/ReplacementInit](https://wiki.ubuntu.com/ReplacementInit)

~~~
icebraining
I never said or meant that Upstart wasn't younger than those alternatives.
What I disagree with was the idea that there was much apathy and then a sudden
crisis when systemd appeared. The development and adoption of Upstart and
OpenRC (even if the latter isn't an init daemon, it's still an alternative to
the init _system_ ) by some of the biggest distros contradicts that claim.

~~~
digi_owl
And i don't think many would have noticed the existence of systemd, as was the
case with upstart, if the projects devs had not started lumping all manner of
non-init projects into the systemd bucket.

My first encounter with the existence of systemd was while keeping half an eye
on the whole consolekit+polkit+udisk rigamarole i apparently needed to get
thunar (or more correctly gvfs) to automount stuff.

This when i learned that consolekit was to be replaced with logind, and logind
required systemd as init.

That, to me, was a very WTF moment. It basically made a file manager dependent
on a specific init being used.

