
Spotify migrate 5000 servers from Debian to Ubuntu - lmedinas
http://raphaelhertzog.com/2014/07/16/spotify-migrates-5000-servers-from-debian-to-ubuntu/
======
tenfingers
Ubuntu LTS is a joke, really. Any major issue found in any core package is
usually forwarded "upstream" (which is Debian).

Very few people in Ubuntu have, apparently, the knowledge to debug issues
which go beyond simple repackaging of an updated copy. Even issues related
strictly to Ubuntu (such as mount dependencies with upstart with complicated
setups with multipath) do not get fixed if the issue is not resolved in newer
releases and/or got already debugged in Debian.

This is not what "support" in "long term support" is. Predictable release
cycle in these terms is not worth much.

For desktop systems it might be different, but then again in our lab (~50
users) most of our Ubuntu's users shifted away to Mint over the course of 3
years. Only one is still using Ubuntu.

But of course I'm biased, I'm a Debian developer myself.

~~~
Alupis
At my company, we don't even consider Ubuntu as a viable server platform. It's
RHEL/CentOS for just about everything.

As you mentioned, the LTS Ubuntu versions are really not on the same level as
the stable RHEL releases and support that is provided.

Yes, Ubuntu is very popular... in both the Desktop and Server arena (mostly
catalyzed by EC2 making Ubuntu the default Linux image, even though EC2 is
built out of RHEL/CentOS boxes). However, most "big metal" servers are
RHEL/CentOS or something else.

In any event (and distro-fany-boyism aside ;-) ... one has to question the
change from Debian Stable to Ubuntu LTS. The only thing I can think of, is
they wanted a more "proper" company to call up for support... even though
Debian are the guys Ubuntu calls up when _they_ need support.

At the end of the day, my 2 cents would be to get a distro that focuses on
being a server distro first and forthright, not a distro that is a jack-of-
all-trades.

~~~
functionalfish
One reason they switched is because of Docker issues on Debian. Source is here
(@4:46)
[https://www.youtube.com/watch?v=pts6F00GFuU#t=284](https://www.youtube.com/watch?v=pts6F00GFuU#t=284).
Whole video is worth a watch though.

~~~
Alupis
RHEL 7 (and now CentOS 7) has native Docker support built in... seems that
would have been a good choice of distro if they were looking for good, long
term Docker support.

------
jgrowl
We're still running Debian Squeeze in production at one of the places I work.
It had one advantage recently: The SSL package was so old that the heartbleed
vulnerability hadn't yet been introduced, lol!

~~~
zorpner
I have a client who was in the same position. They actually still have a few
boxes on Etch... and amazingly, one Sarge box:

$ cat /etc/issue

Debian GNU/Linux 3.1 \n \

They're all running a very old hacked-up version of torque that no one
remaining at the company can do anything with. At least they're walled off
from everything else, I suppose?

------
LaSombra
One thing I don't understand is why companies don't pay Canonical for support?
Is it such a bad thing to pay for Linux support?

I wonder if companies that have chosen Ubuntu LTS think they are supporting
Canonical and its developers just because they are running Ubuntu. This
article doesn't mention Canonical at all.

Support Debian developers is a great thing but Canonical probably spends tons
of money and I rarely see companies saying "I chose Canonical and Ubuntu
because...".

DISCLAIMER: I work for a company that provides Linux support services

EDIT: Typos

~~~
opendais
> One thing I don't understand is why companies don't pay Canonical for
> support? Is it such a bad thing to pay for Linux support?

I've never needed paid professional linux support. Or windows support. Or mac
support. Or android support or any of that.

I think the issue is...once you develop enough in-house Linux knowledge you
don't really derive a benefit from paying for support.

Tbh, the only kind of "linux support" I'd really want is overnight sysadmin
support so I don't get calls at 3am. I suspect that isn't the kind of support
you have in mind.

~~~
rlpb
All software has bugs, and professional end users often hit edge case bugs
that affect few others.

Do you push fixes for these back into your distribution (or upstream and
cherry-pick into the distribution, etc), or do you maintain your own private
repositories containing your modifications?

If you do the latter, then you have extra maintenance costs. If you pay for
support from a distribution vendor, then you save that cost. Even if you have
the expertise to do everything yourself, it still costs you time (and
therefore money). Working with a distribution vendor amortizes that time
between all of their customers, saving everyone money.

~~~
opendais
> All software has bugs, and professional end users often hit edge case bugs
> that affect few others.

I'm kind of amused by the implication that I'm not a professional end user.

> Do you push fixes for these back into your distribution (or upstream and
> cherry-pick into the distribution, etc), or do you maintain your own private
> repositories containing your modifications?

Or maybe, y'know, a third option where one picks software that is highly
reliable for your use case.

You are suggesting a very false choice. If you have unique needs that
frequently cause you to encounter edge case issues, you need someone in house
who can fix those on your timetable and not a vendor's.

------
SudoAlex
As someone who admins a bunch of Debian powered servers, it would've easier
for me if the Debian LTS project had been announced last year instead of
earlier this year.

The 1 year gap between the release of the new version and the end of support
for the old version just wasn't long enough. Sadly I'm also switching servers
to Ubuntu because of it - along with a push towards using Docker containers
powered by distros with long term support.

------
skywhopper
My team switched from Debian to Ubuntu a few years back for precisely for this
reason: a predictable update schedule. Dealing with support isn't really my
concern. I'm able and willing to workaround whatever edge-case issues arise,
but when I deploy an OS on a box I need to know how long I can plan on the
core OS elements being updatable. I need to be able to schedule mass OS
upgrades months in advance. Ubuntu provides that capability, Debian has not.
Though it sounds like Debian has woken up to this issue and is finally
addressing it.

------
trustfundbaby
TIL Spotify has 5000 servers ... somehow I expected they'd have more.

~~~
hedwall
They have more. A guess would be about the double.

~~~
trustfundbaby
I see. Did you used to work there?

Why were the other servers not upgraded?

~~~
hedwall
I'd love too, but no. But I have a close friend that works in their SRE team.
I also meet lots of Spotify people at meetups and such as I live in Stockholm.

My guess would be that they will not migrate their data-related machines
(Postgres, Cassandra etc) as the reason for going to Ubuntu is partially for
Docker and using Docker in their testing and staging environments (to begin
with).

The other one being that they don't have anyone maintaining an patched inhouse
kernel for debian anymore (or they don't want to do that).

