
Dear Package Maintainers: Stop Being Clever - ajdecon
http://www.standalone-sysadmin.com/blog/2013/07/dear-package-maintainers-stop-being-clever/
======
bashtoni
I posted this as a comment on the blog, but I'm duplicating here:

RHEL and Debian (and their respective derivatives) have different philosophies
on packaging, which is why you see this result.

The RHEL philosophy is simple; package installation only installs the package,
starting daemons is a separate process.

The Debian philosophy is that installing a daemon should also make that daemon
run, unless it's not possible the make the daemon run securely by default.
Things like the /etc/default RUN="no" aren't the package maintainer trying to
be clever, but rather them following Debian packaging guidelines. There also
the reason you see things like MySQL listening only on the loopback interface
by default on Debian.

One you understand this the decisions seem a bit less crazy.

For me, the big issue with the Debian approach isn't on installation, but
rather on upgrade. On Debian, running 'apt-get upgrade' on a live server can
cause an unspecified amount of downtime. If you're upgrading MySQL, MySQL will
get restarted. If you're upgrading a library on which multiple daemons rely,
they will also get restarted. yum upgrade on RHEL doesn't suffer from the same
problem, and you can schedule the restart yourself.

Again, this is because of a philosophical difference. On Debian, the idea is
that if you've asked to upgrade something, it should be upgraded. On RHEL, you
can upgrade a package, but continue to run the old (potentially insecure
version).

~~~
Shish2k
> If you're upgrading MySQL, MySQL will get restarted.

Or specifically, it will get stopped, your packages will spend 15 minutes
updating (or longer if there's an interactive prompt mid way and you aren't
paying attention), and then it will get started again.

I'm quite happy with the debian way of doing things, but I'd really like it if
apt-get was smart enough to stop / upgrade / start individual services, rather
than stop-all / upgrade-all / start-all...

~~~
viraptor
15 minutes? Mysql upgrade and restart takes up to 5 second in a usual case.
Also services do get stopped / started one by one, not all at once.

~~~
saurik
It depends on the upgrade sequence; I've definitely seen the behavior of a
service getting temporarily shut down during something like preinst upgrade, a
bunch of packages being installed, and only at the end during configure them
all being restarted.

~~~
viraptor
Report a bug then, or check if this is really possible to restart the service
after installation. In some cases that may be the only reasonable way to do
it. For example services based on scripts/libraries which get dynamically
reloaded on each request cannot be updated while they're running, because it
would cause mismatch between the components. Killing them, updating all files
and starting again is the only way to do it.

As you said - it's not a debian-specific thing. Some packages may do it if
they explicitly require it (and I haven't seen one doing this in the official
repos so far).

~~~
saurik
I did not and do not consider it a bug; it was just inconvenient; it was
really really inconvenient once when I did an upgrade involving hundreds of
packages.

------
txutxu
What I don't understand is why every maintainer adds a (unneeded) layer for
enable/disable.

If you want to run a service, you have a runlevel editor.

This is, chkconfig in RH based, and update-rc.d in debian based.

That is the tool for the job.

Nowadays it seems that everybody prefers to invent an additional switch, this
is a shell boolean or string, in /etc/sysconfig (RH) or /etc/default (debian).

Why ?

------
belorn
When packaging a daemon for Debian, the maintainer should add sufficient
questions during installation, and sane default values, to start the service.

Contrary to the article, I think this is a rather good policy. It makes sure
packages are usable as they are, rather than the normal process of config,
make, make-install, and last adding user configuring before anything will
start up. Debian is not Gentoo.

For those daemons which this isn't possible, package maintainers has a harder
time. One of the better ways is simply to have it run, but in a safe way which
do nothing. That would solve the issues the article describes, and would not
be harmful.

~~~
stephen_g
What does packages using non-standard hacks that break puppet scripting have
to do with configuring and building source code? That's not related to the
issue at all...

~~~
belorn
The article talks about two issues. One is that debian has policy to default
run daemons after installation. The second issue the article brought it up is
the issues with puppet.

Regarding the first issue, sane defaults and usable packages directly after
installation came as a result of the config, make, and make-install process.
Users should get a useful package directly without needing to spend days
adding all the default settings to start a simple mail or web daemon.

The second issue has nothing to do with config, make, and make-install. It has
to do with non-standard hacks. If nagios3 had simply started up like apache,
this would have been a non-issue.

------
reidrac
Every package maintainer needs to make some decisions when packaging software.
Debian has very specific guidelines about packing, so if you don't like it may
be Debian is the wrong distribution _for you_.

Recently a workmate installed nginx from the upstream repo and... surprise:
nginx packages things different to Debian. You may like it more or less, but
there are different solutions for the same problem.

~~~
MPSimmons
Author of the article here. Thanks, and I agree with you. I have some
disagreements with the way Ubuntu does things like this, and I'd prefer an
RHEL clone (which, of course, has tradeoffs too).

That being said, I'm in the environment relatively recently and it's
established, and I'm the one who has to learn to adapt since I don't see our
infrastructure changing dramatically soon. Other than finally adding
configuration management, that is.

------
codezero
I remember installing Red Hat 4, going on IRC and getting rooted. Red Hat's
approach comes from lots of learning, and I think it's smart. Thanks for the
post.

~~~
antihero
Could you elaborate as to how this happened?

~~~
jlgaddis
Years ago, Linux distributions defaulted to installing everything and the
kitchen sink and enabling all those services automatically.

After years of "OOTB pwnage", they started to make things a bit more secure by
default and they eventually got to the point we're at today (where most
installations are relatively secure by default, OOTB).

(Side note: I believe it was RHL 5.2 that shipped with wu-ftpd enabled out of
the box. By the time it shipped (or shortly afterwards, I forget which), there
was a remote root exploit for wu-ftpd. Thus, if you installed RHL 5.2, you'd
likely get pwned in short order.)

~~~
codezero
This is exactly correct. I was a Slackware zealot for a long time but
frequently tried Red Hat because my boss back then was a big fan. I didn't
actually convert to appreciating Red Hat until I worked there and learned the
advantages of using RPM and other tools included in Red Hat that were absent
in Slackware. In either scenario, and this is still true today, if you plan to
connect your system to the Internet, you need to know your configuration and
care for the system after it is live.

Modern distros help with this a lot by offering rapid package updates,
automatic updating and smarter firewall and security measures. A lot of this
stuff was controlled by very esoteric text configuration files in the past,
which made it hard even for a savvy person to get right every time.

------
lmm
Debian always think they know better than the authors of the software they're
distributing (the worst, but by no means only incident being their key
generation bug). When I maintained a linux program I received twice as many
bugs from Debian users as anything else, because Debian repackaged KDE in a
way that broke its autoconf macros. And anyone who's ever tried to follow a
tutorial for e.g. Tomcat will know how radically Debian rearranges its files.

IMO it's not worth maintaining Debian packages of one's software or supporting
it on Debian. Let Debian have their own packager for it if they want to, and
direct support requests to them.

~~~
Daviey
Perhaps you were seeing more bug reports from Debian users, because more
people were using your software via Debian than elsewhere?

~~~
lmm
Right, but the excess was all people with the same problem. I added a note to
the homepage ("debian users: you need to install libblah-devel, configure will
not warn you about this because of how debian packages KDE") but it didn't
seem to make any difference.

------
stonith
We have a similar problem when deploying Openstack using puppet - all the
services need to be brought up after config that sets things like the central
DB and AMQP locations. It means that a deployed site will always have errors
in their logs right from the start, which confuses customers and makes it
harder to spot when there is actually an error in the deployment tool.

~~~
Patrick_Devine
If you're using Ubuntu, the .deb packages need a lot of work. I've been
playing around with this for the past week and there are all sorts of little
glitches and gotchas. A big problem is that the package maintainers didn't use
debconf, so you end up having to install, set some settings and then bring the
service up.

This is definitely not the debian/ubuntu way of doing things.

If I find some time I'll see if I can repackage things to properly use debconf
and push things upstream.

~~~
Daviey
No, Ubuntu specifically made the choice to avoid debconf for Openstack
packages.

If you are using debconf interactively, it simply doesn't scale - if you are
preseeding - or using juju, puppet or chef - what value does debconf add?

Please don't assume that it is simply wrong, just because it doesn't do x or
y.

The debconf templates will always be trying to play catchup, and the
transition of options between components doesn't make a lot of sense.

It would be really good to know what issues you have identified, as we take
quality very seriously in this area.

However, your help with packaging will be most appreciated. Feel free to give
me a shout on IRC, and we'll go through it.

~~~
Patrick_Devine
I'm not quite sure I'm parsing your sentence correctly. If you're preseeding
with debconf the value is that you got to preseed the correct configuration
options. This means the debs will work with anything, not just the flavour of
the week devops tool. Juju, Puppet and Chef aren't the only games in town.
I'll agree that they are all very useful tools, but you shouldn't need to
write complex scripts/manifests/whatever around them to stand up services.
That should happen out of the box without requiring anyone to jump through 50
hoops. It also means that you can scale from 1 instance all the way up to 5000
instances which is a lot better than optimizing for the 5000 instance case.

I'll definitely give you a shout on IRC.

------
jmomo
I hate to say it, but this is just a rant by a RedHat/CentOS admin who has
started using Debian/Ubuntu for the first time and just doesn't know why
things are done differently there.

~~~
MPSimmons
Author here. You're 99% correct. I'm historically an RHEL-based admin, who has
started using Debian/Ubuntu for the first time (relatively, for about a year),
but I totally get _why_ these things are done. What I deeply disagree with is
_how_ they're done.

Why are there so many non-standard ways that a package can decide to not
start? I really appreciate that the package maintainer is _trying_ to do the
right thing, but it's ridiculous that there isn't _a way_ to do it - there's
the Debian default and then there's whatever the packager thought up.

That's nuts, and needing to manage the same resource twice in puppet isn't a
tragedy or big deal, but it's a symptom that something is wrong.

~~~
sciurus
You keep saying things like "so many non-standard ways that a package can
decide to not start" and "whatever the packager thought up". Can you provide
examples of what you mean?

In your blog post you only mention one way of controlling startup- via an
environment variable sourced from a file in /etc/defaults. This is the
standardized, documented place in Debian for controlling the behavior of init
scripts. Offhand, I can't think of any other ways I've seen packager
maintainers controlling startup. I'd really need to see more evidence before
agreeing that things are "ridiculous", "nuts", or "wrong".

[http://www.debian.org/doc/manuals/maint-
guide/dother.en.html...](http://www.debian.org/doc/manuals/maint-
guide/dother.en.html#initd)

~~~
jsight
The file is a standard, but the contents are not. For the most part, that
makes sense, but it would be nice if there were a few standards there. Eg,
DISABLE_ALL_SERVICES=true|false. :)

More examples here: [http://serverfault.com/questions/526075/fixing-services-
that...](http://serverfault.com/questions/526075/fixing-services-that-have-
been-disabled-in-etc-default-with-puppet)

As it is, figuring out exactly how to do this has to be done for each service.
It's tedious, IMO, though I understand the history of why this is the case.

~~~
sciurus
Sure, I can see the value in making service configuration more standard. E.G.
after getting over the learning curve I've found working with systemd more
pleasant than SysVinit.

I disagree that this is currently really more of a problem on Debian/Ubuntu
that it is on Red Hat. E.G. If I want to manage BIND via puppet on Red Hat,
I'm going to have to care about /etc/sysconfig/named.

------
moe
The author is spot on.

This has been a long-standing bug in Debian which, due to their glacial
processes, will probably prevail eternally.

In theory it is possible to disable the post installation auto-start with a
specially crafted 'policy-rc.d'. In practice you don't want to go there, as it
tends to break in unexpected ways.

------
dmourati
I answered the serverfault post.

[http://serverfault.com/a/526123/76707](http://serverfault.com/a/526123/76707)

I'm a longtime Redhat/CentOS guy recently moved over to Ubuntu. It has
definitely been a bit weird getting adjusted.

One counterpoint to the OP's rant? SSH.

------
GrinningFool
It seems like a lot of these comments are missing the point by discussing
virtues of various packaging systems and auto-start behaviors.

It seems to me that the author was saying that consistency is important -
installed services are expected to have a consistent behavior on a given
platform.

A lot of tooling is built based on this, because that behavior is defined by
the respective systems.

When people force override of these behaviors, it does break things as well as
cause frustration.

The article isn't a criticism of any one packaging system or autostart choice
of a distribution, but rather of individual packagers who impose their own
choices over those of the system they're packaging for.

------
rlpb
Debian's thinking on this is that a user won't install a daemon unless he
wants it to run, and that a package should get everything into an immediately
useable state by default. Therefore, a daemon should be started automatically
when a package is installed.

I agree with the objection: I think that times have moved on, and in reality
today people use configuration management tools and almost always want to
change the default configuration on a server daemon. Therefore, it doesn't
make much sense to start it automatically.

As bashtoni says: this isn't a problem with package maintainers. It's a
problem with Debian Policy.

A stepping stone to a full solution is to arrange a standard to configure the
system not to start a daemon by default, and arrange for packages to follow
that standard. policy-rc.d is just such a standard.

------
Qantourisc
I agree on the autostart.

------
Chris2048
I believe ElasticSearch will automatically connect to any nodes with the same
cluster-name. The package installs it with the default cluster-name. Since it
starts on install, you don't have a chance to change this name before it's
done that...

~~~
dehora
Apache's Cassandra deb that a similar behaviour and has a greater impact than
ES's super-hero node names. A newly started Cassandra node will absent a token
declaration in its yaml file, auto-assign a token. This is never what you want
in a cluster distribution. So the workaround is either to patch the deb to not
autostart or let it start, stop it, and blow away the data directories. This
allows cluster management tools (such as Priam) to control the token
allocation first, and then re-start Cassandra.

This is effectively how Debian/Ubuntu work. Once you know it, you can work
around it. That said, it would be nice to have packages that don't run their
service.

