
The sad state of sysadmin in the age of containers (2015) - maple3142
https://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html
======
onion2k
_Ever tried to security update a container?_

The whole point of using a container is that you can destroy it and build a
new one easily. The new one should be built using up-to-date packages with
security patches applied (and tested, obvs). Using the 'pets versus cattle'[1]
analogy, patching a container feels like you're treating it like a pet. You
should just kill it and get a new one instead.

[1] [https://thenewstack.io/how-to-treat-your-kubernetes-
clusters...](https://thenewstack.io/how-to-treat-your-kubernetes-clusters-
like-cattle-not-pets/)

~~~
dsr_
The whole point of having a distro with a good reputation is that you can
incrementally outsource your trust to them, one package at a time. OpenSSL
needs a fix? One package update per machine, and if you have lots of machines,
you update that.

With VMs, you can use the same tools you use with any other machine.

With containers, you are responsible for figuring out which ones need which
packages and how to re-build them. Your method is likely to be idiosyncratic,
so you not only need to be your own security team, you need to be your own
packaging maintainer and infrastructure maintainer.

The two extreme versions of this are the neglect model, in which you just
don't care about anything, and the bleeding-edge model, where you
automatically build with the latest versions of everything pulled from their
origins. Both ways end up with unexpected security disasters.

~~~
regularfry
The "one package at a time" model assumes you care about per-instance uptime
and bandwidth costs. The "reset the world" model assumes rebuilds and reboots
are cheap enough not to have to.

I don't think either is necessarily wrong; there's not a huge distance between
`apt-get install unattended-upgrades` and a daily (weekly, whatever) container
rebuild except that you need to cron the latter. Unless you're using a
different OS between the two cases, they can use the same package database,
the same tooling, and so on.

~~~
dsr_
You need to make the decisions, keep track, and do the work. If you've got the
tools made for you, excellent. Otherwise you also have to build and maintain
the tools -- which is my point.

~~~
regularfry
There are no more decisions to make one way or the other. If you're keeping a
VM image updated with new packages, that's either a) automated with something
like unattended-upgrades, or b) a manual scan of DSA's with decisions as to
whether they're important enough to apply and to which boxes they need to be
applied. It you're rebuilding a container image, that's either a) a 2am cron
job to do a blind rebuild that captures the latest package versions anyway, or
b) a manual scan of DSA's with decisions etc etc...

The 2am cron job, if you go that way, is, assuming you've got an automated
build anyway, a one-line bit of config somewhere. It's not something I'd
usually think the words "build and maintain" would apply to.

------
cuillevel3
> I’m not complaining about old-school sysadmins. They know how to keep
> systems running, manage update and upgrade paths.

Sadly these ways of olde didn't scale. They could only do it for a handful of
servers, change request took months and the systems were not to be 'touched'.

> And since nobody is still able to compile things from scratch, everybody
> just downloads precompiled binaries from random websites. Often without any
> authentication or signature.

Right, that would have never happened back then, when packages were not signed
and freshmeat.net was still a thing. For some reason compiling C code from
some website (`wget;./configure;make;make install`) seems to be more secure
than `go install`, maybe because nobody understands automake anymore.

~~~
zeveb
> They could only do it for a handful of servers, change request took months
> and the systems were not to be 'touched'.

Those old ways of doing things were indeed slower than they needed to be in
many cases, and I'm glad that we sped them up. I don't believe that we needed
to throw the baby away with the bathwater though.

The problem now is that we build systems which are unmanageable and unmanaged.
'Throw it in a container and let it run' is not scalable either: it scales
neither security not maintainability. Instead of scaling, it just _punts_.

Enabling one to do something one shouldn't do (i.e., deploy insecure software
into production) is not a virtue.

~~~
cuillevel3
After a few decades it's no longer a baby ;)

Software is becoming more complex and moving faster than ever. 'Controlling
your dependencies' was always an illusion and a trade-off at best. Often a
trade-off against security, inheriting the folder of JAR files from your
predecessor and such.

I think of containers and all these new tools as designs which are supposed to
help us manage existing complexity. They don't create it, it's already there
in the requirements and real-world deployments. The friction we feel, is that
some of those tools are not very good (yet), but maybe better than the
previous generation (sometimes). On the negative side, there is a lot of
nostalgia ("ah, do you remember bare metal...") and unwillingness to change
and learn.

~~~
wayoutthere
I think a lot of folks who have been through a few tech cycles just see us
building the same tools over and over again adding complexity to the overall
system.

I think the main cause here is, ironically, the fact that businesses keep
trying to remove sysadmin (as a discipline) from the IT value chain and
replace it with subscription services. Sure, you _can_ do it, but at some
point things will break and you need someone who understands what is happening
at the lowest levels of abstraction.

------
lmm
You've got to look at this in a context where platform package managers like
apt are simultaneously 1) platform-specific 2) jealous, insisting that every
language has to conform to their way of doing things and 3) fundamentally not
very good, having very limited ability to do things like install packages for
a single user or install multiple versions of the same package. Platform
package managers like maven have been far more successful because they provide
a much better user experience, and frankly I suspect this is if anything
_because_ they're so thoroughly isolated from the platform package manager.

~~~
rumanator
> jealous, insisting that every language has to conform to their way of doing
> things

Let's ignore for a moment all your baseless assertions and focus on the
following question: how do you ensure that a build is reproducible if you have
no control on which required dependencies are downloaded by a build system
which requires root access?

Is jealousy the only explanation that comes up to you?

~~~
lmm
Reproducible builds were a non-goal for apt for most of its existence.

~~~
rumanator
> Reproducible builds were a non-goal for apt for most of its existence.

Your comment sounds very disingenuous, as Debian was first released in 1993
(about 26 years ago) and has been pushing for reproducible builds since 2000.

[https://wiki.debian.org/ReproducibleBuilds/History](https://wiki.debian.org/ReproducibleBuilds/History)

~~~
lmm
Your own comment is extremely disingenuous. Your own link states that there
was very little enthusiasm for reproducible builds when mentioned in 2000 and
2007, and the first serious effort towards reproducible builds started in
2013.

------
jve
This all boils down to trust.

`curl | sudo bash` is no different than .\install.exe. The question is about
trusting the SOURCE and trusting the DISTRIBUTION channel (that HTTP download
from scala-lang.org violates this).

Where did you get it? from
[https://microsoft.com/..](https://microsoft.com/..). or from
[https://micro.soft.com/...](https://micro.soft.com/...)? Whom you trust more?
The same with pre-built VM image or whatever... do you trust the party that
made this image/container available?

~~~
LockAndLol
`curl | sudo bash` should never have become a thing in the first place. To me,
it's the most "WTF?" mode of installing anything. __Especially __when the
source is HTTP.

In what world is that secure?

Pity Qubes is so heavy, otherwise I'd be using it

~~~
eejjjj82
would you feel better if your Makefile looks like this:

    
    
        install:
            [ $(id -u) -ne 0 ] && echo 'please run sudo make install' || curl https//my.thing | /bin/sh
    
    

it goes back to what jve said - it's a matter of trust. how often do you
blindly run 'sudo make install' w/o reading the entirety of the build script?
all the time I bet - it's because you trust the source

~~~
LockAndLol
No.

I think I have never run `sudo make install`. Things I install come from
package managers. Docker or a VM is used to test other software.

And of course trust is important, but if I encounter things like `curl |
bash`, my trust is lost.

~~~
yori
What difference does it make if you run `make install` or `curl | bash`? In
both cases, you are running code you have not audited yourself. Or are you the
kind who inspects every Makefile and installer script before installing? If
not, why is one better than the other?

~~~
sudosysgen
Because at least with make install you can have some trust in the delivery
method, ie encrypted git or whatever. If you use just plain http....

~~~
yori
So wget
[https://example.com/installer.tar.gz;](https://example.com/installer.tar.gz;)
tar -xvzf installer.tar.gz; cd installer; make install is okay for you?

But curl [https://example.com/installer.sh](https://example.com/installer.sh)
| bash is not? Why?

~~~
sudosysgen
I'm fine with the second too as long as the host is trustworthy and it's
always https. Most of the time it isn't.

------
1337shadow
The holy grail of sysadmin has always been deployment of immutable images,
instead of host system pollution, source code mutation during deployment ...

It was doable with things like Packer, KVM or OpenStack, now it's become
easier with containers. You can still orchestrate your stuff with ansible or
bash scripts.

I most certainly remember how I was doing isolated deployments 10 years ago:
[https://github.com/jpic/bashworks/tree/master/vps](https://github.com/jpic/bashworks/tree/master/vps)
HINT: it's much cleaner and much more automated with containers: with GitLab-
CI it's trivial to host a private container bulding infrastructure with a
registry and other automation ...

------
axegon
Talking from a developer's point of view, I've thought about this for a while
and I think it has something to do with what I refer to as "anti-imposter
syndrome" \- the notion that anyone can make software. You see, over the past
10 years or so, self-proclaimed "educational" services/websites/institutions
have been shoving down everyone's throat the idea that anyone will be able to
create the next FAANG from scratch after just 3 months of training. Which is
the same as claiming that I can become an F1 driver in 3 months. Where do I
sign up?!?!?! The thing is, F1 results are a lot more visible since you have a
point of reference - the top dogs. Chances of scoring a fastest lap or coming
even close to them - little to none. Software - not so much - you have a shiny
interface and what it does underneath is always a mystery to the end user. And
people who have gone through those magical 3 months of training are often lead
to believe that the way they are doing things is what everyone is doing and
that is how it should be done. Often people who have signed up for those
courses are people who have just a smudge above average technical knowledge -
they have no idea how OS'es work, what should be considered safe or even why.
Don't hate me for saying it, but essentially Windows users. And this is the
software they end up building and distributing. 10 years down the line,
thousands have picked up random scraps of knowledge from here and there and
tried to mash something together. Don't get me wrong, I think technologies
like Docker are astonishing and an incredible tools in the hands of people
with knowledge and experience. However way too many people have picked up some
scraps from there and created the unholy mess the author is talking about.

------
jwr
I recently had a similar discussion with people using npm for building a CSS
framework library. I tried to explain the concept of getting a pre-downloaded
tarball and using "make" (or similar) to produce target artifacts from source
files in a deterministic, repeatable and reliable manner, without relying on
any third-party servers being available and without pulling in dependencies
that might have changed.

It seems the concept was entirely alien to programmers younger than me.

~~~
ekidd
> _without relying on any third-party servers being available and without
> pulling in dependencies that might have changed._

There are two different issues here:

1\. Not pulling in changed dependencies. This is what "lock files" are for: To
limit builds to known version of every dependency. npm was terrible about this
for a long time. Most other language package managers are better.

2\. Not relying on third party servers to be available. Personally, I've
mostly worked with Ruby's bundler and Rust's cargo, and in over 10 years, I've
lost maybe two days of work because of package server outages. That's less
than I've lost due to S3 outages, less than I've lost due to broken backup
systems, and less than I've lost due to complex RAID failures. For the clients
and employers in question, this was acceptable downtime.

In the rare cases where one day of noticeable build downtime every 5 years is
unacceptable, then it's usually possible to just "vendor" the dependencies,
usually by running a one-line command.

For many small to midsize businesses (and many growing startups), a small risk
of brief outages is an acceptable engineering trade-off.

~~~
hn_throwaway_99
> npm was terrible about this for a long time.

That's putting it lightly. To me "terrible" would mean they just didn't
support it, "batshit insane" means they supported lock files but ignored them
every time you ran npm install.

My favorite comment from this stackoverflow post:
[https://stackoverflow.com/questions/45022048/why-does-npm-
in...](https://stackoverflow.com/questions/45022048/why-does-npm-install-
rewrite-package-lock-json)

"Why would you expect something called package lock to lock the packages?
Package lock is analogous to how, when you put any key into a door lock, the
lock reshapes itself to match whatever key was put in, then opens the door.
Now if you'll excuse me, I'm late for a tea party with a rabbit."

~~~
hinkley
I think one of the scenes with Humpty Dumpty could be an appropriate allusion
as well.

------
jillesvangurp
The maven/gradle ecosystem actually does use signatures. That's one of the
things they got right. Maven central has no unsigned artifacts. Also, the
signatures are actually checked on download. Other maven servers may be a bit
more sloppy of course. In any case, if you are running that on a production
system, you are doing it wrong. You run that as part of CI. The output of CI
is containers which you start on a production machine.

The point about containers is that they should be immutable. Running
chef/puppet in kubernetes is not a thing (I hope, probably somebody went
there). Updating a container means replacing it with a new one, that hopefully
you or somebody else built in a responsible way.

That last part is indeed a problem that we have moved instead of solved. 20
year ago, people were just sticking whatever they downloaded on hardware they
bought at the store and banged on it until it worked (very literally
sometimes). So, I guess this is progress but indeed hardly ideal. I remember
using puppet. Can't say that that is with any level of fondness. I vastly
prefer Dockerfile and having CI systems produce containers from those.

Installing things on a filesystem is no longer that common at deploy time
unless that filesystem is that of the container you are building using a
Dockerfile or you are self hosting kubernetes (aka. reinventing a lot of
wheels at great cost). Puppet/chef etc. are still of use if you are doing that
but otherwise it has a limited role in IAAS type architectures. The closest
thing is perhaps pre-baking AMI images using packer and some tool like
ansible, which is nice if you want to avoid having a lot of startup overhead.

Hadoop is complicated, which is why companies exist that host that for you or
will help you hosting it on premise in a responsible way. If you want to DYI,
you indeed have to do a lot of things and do them properly. Kubernetes has
moved that space forward in recent years; so it's easier but this is not for
everyone. If on the other hand you are messing with puppet to get this stuff
done, maybe reflect on the wisdom of not standing on the shoulders of giants
rather than blaming the internet for your self inflicted pain.

------
buzzkillington
Looked at the Debian wiki the OP linked to:

>Debian currently does not include Hadoop packages. There are a number of
reasons for this; in particular the Hadoop build process will load various
dependencies via Maven instead of using distribution-supplied packages. Java
projects like this are unfortunately not easy to package because of
interdependencies; and unfortunately the Hadoop stack is full of odd
dependencies (including Apache Forrest, which for a long time only worked with
Java = 1.5).

>If you want to build Debian packages, the most complete efforts can be found
at the Apache Bigtop project
[http://bigtop.apache.org/](http://bigtop.apache.org/) . __Unfortunately, the
build process for these packages is currently of a disastrous quality, and
should only be attempted within disposable virtual machines, as it requires
root permissions and will install non-packaged software. __

> These will allow building Hadoop packages on a recent Debian system.
> __However, the resulting packages do not live up to Debian quality
> standards. In particular, they include copies of .jar files from other
> packages, and will thus not benefit from security updates done to these
> packages. __

> If you are interested in getting Hadoop packages into Debian,
    
    
        Coordinate with the Java packaging team
        Coordinate with upstream Apache Bigtop 
    

>to avoid duplicate efforts. Thank you.

[https://wiki.debian.org/Hadoop](https://wiki.debian.org/Hadoop)

Jesus Christ.

~~~
alfozan
Who still uses Hadoop anyway?

[https://spark.apache.org/](https://spark.apache.org/)
[https://www.iguazio.com/data-science-post-
hadoop/](https://www.iguazio.com/data-science-post-hadoop/)

~~~
buzzkillington
Spark is worse because you need Scala as well as regular Java.

I've tried building it for my day job, I would rather have a colonoscopy
without sedation.

It's more pleasant and dignified.

~~~
itg
And then add PySpark on top of that. Couldn't leave my last job fast enough
when they decided to use Hadoop/PySpark when the largest incoming files we
received were at most a few GBs.

~~~
boxy310
I once had a consulting gig where the customer desperately wanted to build a
Spark/Scala ML pipeline, for a dataset that was 10 MB. We spent 3 months
hammering it together for a flat Python process that would've taken us 2
weeks.

~~~
snaky
> This find xargs mawk pipeline gets us down to a runtime of about 12 seconds,
> or about 270MB/sec, which is around 235 times faster than the Hadoop
> implementation.

[https://adamdrake.com/command-line-tools-can-
be-235x-faster-...](https://adamdrake.com/command-line-tools-can-
be-235x-faster-than-your-hadoop-cluster.html)

------
majewsky
Discussed at the time (443 comments):
[https://news.ycombinator.com/item?id=9419188](https://news.ycombinator.com/item?id=9419188)

And again in 2018 (426 comments):
[https://news.ycombinator.com/item?id=17083436](https://news.ycombinator.com/item?id=17083436)

~~~
mike_hock
But unfortunately, it's relevant to this day.

~~~
majewsky
More than ever.

------
pram
Terrible article IMO. Hadoop is an awful mess, but it has nothing to do with
Docker, which is simplistic in comparison.

"Ever tried to security update a container?"

Yes, I have! In fact you can maintain patch compliance in a container pretty
much the same way you'd maintain a VM or bare metal Linux installation!

~~~
tomtomtom777
The article focuses on Hadoop, but the generalization to containers is spot
on.

Containers have become the first stop method to hide overly complicated build
processes. When you've noticed your wiring has become a complete mess of
knots, just put it in a box so no one will trip.

It seems to me that containers are useful for deploying final setups, but
should be a no go for packaging tools.

~~~
indigochill
>When you've noticed your wiring has become a complete mess of knots, just put
it in a box so no one will trip.

On the other side of the coin, putting stuff in boxes is the essence of
architecture. CPU opcodes? Boxes. Compiling readable code into bytecode?
Boxes. Objects? Boxes. Functions? Boxes.

Sure, it can be abused. There might be spaghetti in those boxes. The spaghetti
should be chopped into digestible chunks (smaller boxes!). But putting it in a
box makes that an implementation detail. Putting stuff in boxes is still
progress.

~~~
bluGill
All problems in programming can be solved by adding another layer of
abstraction (ie a box) - except too many layers of abstraction.

Good architecture is not about putting things into boxes, but about figuring
out what boxes you need and which you do not. Sometimes a box is good,
sometimes a box is bad. It isn't the fault of the box that it does't contain
the right thing.

------
_wldu
Standard Makefiles are really under appreciated today. They are simple and can
be used with most any language, version controlled and have been around for
decades.

~~~
outadoc
They don't solve any dependency management problems.

~~~
dfox
Most ports like systems actually use makefiles to solve cross-package
dependency management. (Whether that is good approach and whether writing what
is essentially package manager as hairy ball of makefiles is good idea are
another questions)

------
upstandingdude
ha +1 for the build tool mess. its like everyone goes "duuude this stuff is so
complex lets build something on top to abstract it all away" and you end up
with another level of complex stuff that builds the stuff that builds the that
wraps the stuff that you originally wanted to use...

------
rb808
I agree the last decade has been really tough for sysadmins, but the payoff is
developers have had a great time and I dont have to rely on those guys any
more.

~~~
opsthrowaway0
Flash bulletin from the engine room: you're still relying on "these guys" (and
gals), you're just cost-sharing our salaries across whatever pile-o-garbage
SaaS you convinced your architect to engage, including my own large cloud
provider. Hint: it'd be more effective (and cheaper) to employ us directly,
but you do you.

That said, at the end of the day, I'm still sitting in my Aeron and your
"great time" just means concentrating an entire discipline of professionals
into cloud companies and ISPs. In this dream scenario (for who?), you get to
sit around waiting for ticket updates while your revenue craters instead of
picking up the phone or using that hole on your face to communicate with a
peer. I get to focus on systems at scale and on monitoring and metrics instead
of an annoying developer who escalates to their SVP and makes an inter-org
ruckus when their TLS certificate expires and they can't find the error in
their voluminous Javascript "logs". To be clear: I'm having a great time.
There's a suspicious TAM in between me and you now, and you don't have my
extension. I prayed for this day across many developer escalations and _still_
can't believe it's here.

Seriously, people who are likely replaceable with Microsoft Access and a few
Thoughtworks clones shouldn't throw stones at the professions completely
supporting their Blue Bottle habit.

Oh yeah, the Internet keeps on quietly working, too. Tip jar is to your left.

~~~
finnthehuman
All the claims that cloud computing is some boon to developers makes me
suspect they might just work at a terribly organized company.

If you make developers pull double duty as sysadmins, then outsourcing surely
makes things better for development. But it's still more work than handing the
infrastructure and administration over to people that specialize in that. A
well managed service is a ticket away, not getting someone who "knows docker
pretty good" to have another background responsibility to neglect until it
sets on fire.

Throughout the cloud boom I've worked at a company with a great IT department,
not just a glorified helpdesk. The division makes sure that we provide a sane
deliverable that doesn't need constant handholding and 10 hotfixes a day, and
they give us the peace of mind that it's not going to go sideways. They sweat
the details so much that the rare issues that have escalated to the head of IT
and myself at the same computer investigating something were still relatively
minor in the grand scheme of things.

------
Havoc
And it seems to be getting worse lately with everyone talking about their
cloud-y build "pipeline". Which invariably seems to involve stringing together
a bunch of different service offerings from startups that might not be here
tomorrow.

It's hard to imagine something more fragile

------
disordinary
I don't agree.

Systems are more complicated to build with a lot of dependencies that's true,
but this is mitigated by shipping pre-built binaries and containers which have
the dependencies bundled. Yes, those binaries can be from untrusted sources
but if you use those you're taking a risk - the same as if you use code from
an untrusted source (just because it's not compiled doesn't make it safe, when
was the last time your organisation audited anything), ultimately a business
should be using banaries from businesses they trust.

Containers actually increase security in the following ways: 1\. They run
isolated from the host system and if you set things up properly they should be
non root. 2\. They're immutable and cannot be tampered with. 3\. Because they
work on layers you can change them without affecting the underlying container,
to upgrade you can just replace the layer that you inherit from, simplifying
the upgrade path (and being immutable, stateless, and versioned there's no
risk to upgrading, you can very easily rollback). 4\. In the example listed
above like Haddoop you can install the outdated version of Java that it
depends on without compromising the host or other workloads on that server.

Of course, this all breaks down if you don't use containers from trusted
sources and if you don't have processes in place.

It works in a modern environment with pipelines and automation, it doesn't
work as well in a more traditional environment with sysadmins hand wrangling
deployments.

Of course the most secure way to do things is to have deterministic builds of
a codebase which creates a hash which you can compare with a canonical source,
but this is tricky and not all build toolchains support this properly so
you'll end up with mixed results. For a streamlined business the best
compromise is to get applications from trusted vendors and rely on static
analysis and the ability to quickly upgrade if a vulnerability is discovered.

Ultimately, the one of the best ways to secure a system is to establish an
unbroken, audited, and automated trust chain and remove the ability for human
intervention.

------
AndyMcConachie
Things will break and I will need to be able to debug them.

New versions will be released and I will need to be able to upgrade to them.

Bugs will be found and I may need to know if I or my users are experiencing
them.

Security holes will be found and I will need to either upgrade, or mitigate
them in some other fashion.

All of the above realities mean I want well defined and understandable
software. The less well defined and understandable a deployment is, the less I
am interested in deploying it. Complexity is a cost, and this cost cannot be
waived by simply chucking everying into a container.

Simplicity means cheap. It means time not wasted and it means fewer headaches.

------
rbc
The only part of this article that I really disagree with is the title. The
main issue here is release engineering, or rather, the lack of it. Software
development and system administration have well developed traditions, but
release engineering doesn't seem to get much attention in comparison. Given
that supply chain attacks are now all the rage in the black hat community,
maybe release engineering will start to get the attention that it deserves.

------
Russtopia
Amen. I avoid any container-based or blindly 'curl |sudo' install (the latter
only after manual inspection of such a script).

Apps which only offer containers are untrustworthy black boxes. And it
indicates devs who are too ignorant and/or lazy to make even basic efforts at
distribution-neutral or portable code.

Gentoo and Funtoo keep devs honest to some extent, as packages must be built,
not just slurped down in whatever alien form the project decided to use.

~~~
james-mcelwain
I'm not sure about other container solutions, but Docker isn't a black box at
all. It's fully introspectable -- I often dump a container's file system to
see what's actually included.

------
Yuval_Halevi
I saw this comment now from 4 years ago on reddit:

Software packaging and building seems to be becoming more complicated and more
disconnected-- particularly as more specific tools continue to be developed.
It seems every little corner has their own dependency management and build
management solution.

The Docker comments seem a little backhanded. If one looks at a Dockerfile,
there's not much to complain about. It downloads a key, it adds a repository,
installs a package, and some very simple docker-specific tweaks.

If you distrust Docker's signed image, building your own is as simple as doing
a git clone ..; cd docker-nginx; docker build. Docker encourages disposable
containers, separating data, and making your own images. There's still a lot
of work for Docker to do regarding signed images, but I'd argue running
isolated images with documented changesets and simple build files is far
different from blindly running 'curl | sudo bash'

[https://old.reddit.com/r/programming/comments/33ktc9/the_sad...](https://old.reddit.com/r/programming/comments/33ktc9/the_sad_state_of_sysadmin_in_the_age_of_containers/cqlxne5/)

------
IOT_Apprentice
So nearly 5 years later, is any of this any better? Also, why haven't Linux
vendors standardized on a single package manager that is the default for ALL
Linux distros? This seems like the equivalent hell that npm gives to NodeJS.

~~~
mappu
_> why haven't Linux vendors standardized on a single package manager that is
the default for ALL Linux distros?_

The two distro package managers that "matter" (apt and rpm) are similar enough
that this isn't a problem in practice. You can have fpm spit out a deb or rpm
or arch pkg with just one flag change, and host it on your own website.

The real differences aren't just mechanical packaging differences, but more a
question of philosophies, there are a lot of differences in upstream distro
packaging policies themselves. Debian isn't going to let you upload it
directly unless it is a good citizen on their distro, stays up-to-date with
their versions of all dependencies (so that you only end up with one python,
libjpeg etc on your system) and is certifiably free software down to the
bones. Whereas Arch will take literally anything in the AUR, and the fast way
into CentOS is to be employed by RH (paraphrasing).

(I tried to package a certain app in RPM, but it turns out that OpenSuSE and
CentOS have different names for the tzdata package, and until very recently
rpm didn't even have a system for specifying alternate dependencies.)

------
_def
I don't know that much about the state of container systems these days, but to
me it seems it's the "comfortable" way in the "security vs comfort" tradeoff.
Use where applicable and hope for better days.

~~~
arberavdullahu
I don't think you should put security at any case in tradeoff, by your
argument one can save passwords in plaintext because its easy and comfortable,
but its just not acceptable and this would effect not only your app but every
app that your users use.

~~~
ThreeFx
I see the value when running e.g. home instrastructure on some old laptops or
other non-critical stuff. Most people have NAT at home and thus whatever
services they pull and listen in their local network aren't exposed to the
outside world, and they can enjoy mopidy/NextCloud/whatever without going
through a lot of hassle setting it up.

Of course once we come to enterprise environments the opaque build/deployment
process is atrocious.

~~~
BlueTemplar
30% of the Internet already uses IPv6. NAT is obsolete and was never supposed
to be used for security anyway...

------
z3t4
So you dont want to run (insert software here, like your IDE) in the cloud,
and you dont want to run it yourself because you dont trust it (its open
source). Then how do you want it delivered? Snaps? Docker? KVM image?

~~~
disintegore
In an InstallShield package, with a 7-step license activation process.

Let's face it, security is an asymptote. I just want someone I can call and
scream at when we eventually get pwned and I'm ready to pay for that
privilege.

------
ascotan
This post reminded me of an old article:
[https://blog.codinghorror.com/vampires-programmers-versus-
we...](https://blog.codinghorror.com/vampires-programmers-versus-werewolves-
sysadmins/)

The big win for 'containers' is that they separate the concerns of sysadmins
from developers. The days of the sysadmin administering the tomcat and apache
installs are shortly coming to an end. For my part, i think this is a good
thing because devs and sysadmins have different concerns that don't overlap
all the time.

------
kube-system
Using stuff blindly from Docker Hub is a bit sketch, but it’s not a hard
problem to fix. Read the dockerfile before using it, adjust if you don’t like
it, and build and push to your own repo. Then run your vulnerability scans of
choice on them nightly.

------
cosmodisk
"Consider for example Hadoop. Nobody seems to know how to build Hadoop from
scratch. It’s an incredible mess of dependencies, version requirements and
build tools." And that's exactly what's wrong with a half of the software
industry. Some people even take pride in thinking that having dependency hell
is a good thing. A lot of build tools look like they are created by aliens to
aliens with little thought of how the end user would use it. And if someone
dares to say how crappy the whole thing is, they get dismissed for not being
smart enough or simply lack dedication to spend unlimited hours trying to
build some 500 line code.

------
t0astbread
I'm looking at this from a relatively neutral newbie POV. I've tipped my toes
in both administering machines the "classic" way and with tools like Docker
and Docker Compose. To me both have their benefits and drawbacks but Docker
looks a bit better overall. Here's why:

1) The idea of immutable systems rocks. It feels like the logical next step
from containers.

2) This is maybe just because of me but I have the feeling services that can
easily be (automatically) updated (like with Docker) are gonna be more up-to-
date on average than when I have to build every new version myself. (Although
that could be automated probably.)

------
de_watcher
[http://kmkeen.com/maintainers-matter/](http://kmkeen.com/maintainers-matter/)

------
PaulHoule
Uh... I've built Hadoop from source, it is not that bad.

~~~
icedchai
I've also built it from source, admittedly back in 2012. It wasn't that bad,
either! My concerns were building an AMI image, not "packaging" it in a more
traditional sense.

~~~
PaulHoule
I think most of the conventional image building tools are stupid. (e.g. they
take something that could be done in 10 minutes and make it take an hour, are
prone to failure because of too many round trips, ...)

I made rather complex images that included a large database (e.g. 30 GB) and
Java apps, and found the easy way to do it was write a program that analyzes
the problem, writes a shell script that builds a bastion and builds the
software, then put that in the cloud-init... The machine "phones home" via SQS
to tell the controlling server to shut it down and image it.

------
znpy
[https://xkcd.com/1988/](https://xkcd.com/1988/)

------
turk73
Before containers, I had sysadmins who ran our AIX cluster. Three nodes, not a
single one configured like the other. There was zero consistency where it
counted and these jokers would go on a terminal and manually configure
Websphere and other components and screw it up constantly. We even tried to
give them a terminal tool where you could type in one window and it would
mirror on other boxes, but they refused to use it.

Puppet, Chef, Ansible, all led the way to something better. Docker and
Kubernetes feel so far advanced from what we had in 2012.

------
_pmf_
Give me rsync and a sysadmin any day over a circle jerking dev ops team.

~~~
ex_amazon_sde
Amazon and some other FAANGs build and deploy OS packages, and by choice.

Sad to see how anything that is not container gets downvoted here.

------
duelingjello
Yes, it's worse than that on a couple of fronts. Cloudera, Amazon and others
benefit financially from situations of over-complicated messiness, while
average SWEs, SAs and others are left with wasted time, money and lack of
trust. And worse, lax engineering and operations practices introduce attack
surfaces to fraudsters, thieves, organized crime and state-level actors to
backdoor, destroy, steal, extort and monetize infrastructure of organizations
large and small.

------
dig1
The same applies today for nodejs and go, but the same was happening with
python, ruby and perl libraries in the past.

However, this only shows that standard distro packaging approach is broken or
was broken all the time. First, if you want to distribute you application (or
library) via distro packages, you'd either have to build them and publish on
your site or have to go through long mailing process to satisfy distro
specific packaging guidelines and hope that your package will end up in
standard (or extended) distribution. Debian is also notorious for "reviewing"
and modifying source code, without contacting authors (remember Debian SSH
fiasco).

I had, not once, unpleasant experience with this process.

Today, things are much easier - pack your whatever and put on dockerhub, maven
or github. No middleman and no headaches just to reach your users. Sure, there
are drawbacks (like mentioned in article), but we'd have to leverage things
like GPG and proper package signing. Now, that is totally different story,
because cryptography, even the basic one, is PITA for most developers and
users. You can't have both :D

