
Ask HN: How do you manage cross-system dependencies? - perlgeek
On Debian (and many other Linux distros), managing dependencies on a single system is easy. Just declare that package &quot;web&quot; in Version 2.3 depends on package &quot;backend&quot; in Version 0.3.1 or higher. If you try to install &quot;web-2.3&quot; but your &quot;backend&quot; is too old, apt-get&#x2F;aptitude will complain.<p>How do I get the same&#x2F;similar behavior when the frontend and backend are on different servers?<p>When I heard of &quot;orchestration&quot; tools, I hoped they&#x27;d solve that problem, but the products I looked so far look like schedulers &#x2F; pipelining systems, not like dependency managers.<p>What tools do you use to manage dependencies on multiple servers?<p>(In case it makes a difference, we use mostly Debian Jessie, and deploy our software as .deb packages via GoCD. Puppet is used for configuration management, though not for the deployment)
======
spotman
You would architect your application to not be so brittle that deploying one
wrong package takes the service down. For example if you have to make breaking
changes try to make your backend smart enough to handle web2.2 and web2.3.

Tools like configuration management (chef, puppet, ansible, etc) can help you
roll out the different versions , and interrogate servers to see what is
installed on them.

So in the case of rpm/deb, if you need to update a package and it comes with
dependencies I would let the package manager resolve them. But specifically if
your packaging your own application components in this format you would not
have control to tell server A to be dependent on server B's dpkg state. That
would be complicated and brittle.

You want to use rpm/deb/etc to resolve local dependencies only. For setting up
infrastructure that needs supporting infrastructure at a certain version to
work well this must coordinated at some level even if by hand.

These days there is more than several handfuls of different ways to approach
this. I recommend chef or ansible if your new to configuration management.

Edit: oh you use puppet already I see. Just have puppet update packages as
needed, and consider using it for deploys. You can group systems together
conceptually and do all kinds of things even blue/green deploys, rolling
deploys etc. you can use mcollective to see what packages are where.

~~~
perlgeek
Thanks for your reply. It's clear that you've put a lot of thought into it,
but I'm still struggling make it useful for me.

> These days there is more than several handfuls of different ways to approach
> this. I recommend chef or ansible if your new to configuration management.

As far as I can tell, neither ansible nor chef really help me with answering
questions like "can I deploy service X in version Y to machine Z, or are
dependencies missing?". Or do they? If yes, how? Or did I misunderstood what
exactly they should help me with?

~~~
spotman
Lets give a quasi specific example maybe..

lets say you have service X, is your ApiService 1.2, and it's in production.

Lets say you have machine 1 and machine 2, both running ApiService 1.2

In puppet, you could use hiera to know what version of ApiService a machine
should have. You can use mcollective to interrogate these machines to verify
that puppet successfully installed ApiService 1.2 on these machines.

Now, to answer the question "can i deploy ApiService 1.3 on machine 1", I
think that that would be best answered by testing this in a
development/staging environment with another machine that is not serving
mission critical traffic.

Here you would do the testing you need, and ensure it works, both in a
installation capacity (say ApiService 1.3 needs a newer version of libxml2 or
something), and in a general "does it pass QA" standpoint.

It would be your package maintainers job to specify that ApiService 1.3
requires libxml2 version XYZ.

Puppet/etc (or, specifically when dpkg runs via puppet) would fail to install
the package if its not going to have libxml2 version XyZ.

Being that it's your package, you would write it in a way that dpkg is able to
install it on the target servers your targeting. So, you could even spell it
out exactly in puppet to, "before installing ApiService-1.3.deb, install
libxml2-2.x728.deb"

If what your getting at, is that if ApiService 1.3 on machine 1, relies on
libxml2 being updated on machine 2, then that sounds a bit brittle to me and
should be refactored to be more forgiving.

Maybe that helps make it more clear, if in doubt will look for more comments.

~~~
perlgeek
Let me see if I got this one straight:

So puppet has some infrastructure for querying package versions (and other
facts) (mcollective), and a way to store the dependency data (hiera), but the
actual management still has to be done manually (potentially by trial & error
in a dev/test/canary environment).

Right?

If yes, that's good to know, but still less than I was hoping for.

~~~
spotman
Yeah, i look at os-package dependency management as largely a problem that
should be left up to the os' package manager I suppose.

So, naturally just like any change, you would want to test the upgrade before
rolling to prod.

What are you hoping for I guess? Im genuinely curious. Maybe you want puppet
to complain before running that system X doesn't have version C of package K ?

How is that different than it failing on dpkg?

(its possible I'm not understanding..)

~~~
perlgeek
> What are you hoping for I guess? Im genuinely curious. Maybe you want puppet
> to complain before running that system X doesn't have version C of package K
> ?

In the past months we had several failures where new features weren't working
in production, and it was usually due to neglecting to upgrade some
components.

So for example one developer creates a new feature in the front end, a second
developer creates the backend service, notices a missing feature in a second
backend service, and delegates that to a third developer.

When the time comes to roll out the frontend service, the first two developers
probably remember that a new version of the first backend service is required,
but sometimes forget that there's also a dependency on the new version of the
second backend service.

So the feature goes into production, but fails because an indirectly dependent
service wasn't upgraded as well.

We only have one testing environment, not one per service, so we don't easily
notice the problem in testing.

So, if all the services where installed on machine machine, the developers
would just need to declare the proper dependencies (which they can do at the
time they discover them), and an "apt-get install <frontend-service>" would
resolve the whole dependency chain, and either abort the upgrade if the new
version of the second backend service isn't available, or upgrade it too.

What I really want is the same easy of dependency management, even though the
packages aren't all installed on the same machine. I simply want to declare
all of a packages versioned dependencies, and then say "upgrade service X" and
it'll automatically tell me what I need to deploy (or even do it for me).

Currently our workaround is a wiki page in which we document the depenencies,
and manually resolve them when the time comes for a deployment. But this is
very frustrating, because the problem behind it is solved in any decent
package manager -- but only in the single-machine case. So just because we run
more than one machine, we have to reinvent the whole dependency management
wheel.

And it also frustrates me that I can't seem to find much discussion of this
topic, even though I'm quite sure we're not the only ones who have these
problems. Am I just looking in the wrong direction?

~~~
spotman
Thanks for explaining. This is a pain point I think everyone that does service
oriented architecture has to some level.

Most shops I have worked in seem to mostly attack this via procedure. Ie -
release is OK'd by QA, then deploy ticket is made that specifies what versions
go where, and then operations is responsible for executing this procedure and
ensuring its sanity.

If you have a lot of hands in the pot, (too many cooks), then this is bound to
have issues when everyone is busy and things get overlooked.

There is a lot of automation that can help however. Dependency management in
this sense I think has to be more elaborate than just "is X installed or
deployed", but you might also want to ask "is X healthy?" Just because a
package has been installed doesn't mean its working.

So that is in my opinion why there is no one-size-fits-all solution for this.
Many releases are fully compatible with each other, and in simple applications
probably even more so, but once you have many teams working on many services,
it can be a nightmare.

A lot of places have granular feature management. So, frontend can fail
gracefully if feature is not supported on backend. Often backend supports more
than 1 fronted version set. You can use versioned api's (like putting a
version in an http header, or namespacing features like /api/v1/x, /api/v2/x).
(it's been said a huge distributed app like facebook has something like 500
different versions of components all running concurrently)

You can have your deployment automation do things like check to see if backend
has required version (which would be something that you would have to put in
hiera/equiv of what versions are required), and even more importantly if they
are responding to health checks.

One project I currently work on has a very distributed architecture, and every
application has to conform to a health check endpoint that returns its state
and its version. So we have scripts that do either rolling deploys or
green/blue deploys depending on what we are deploying, and we can define the
order in which services are deployed. We can deploy them concurrently/rolling,
or independent/rolling, or either of those flavors non-rolling, or green/blue.

So if we say we are deploying fronted3.1, backend4.2, and we are replacing
frontend3.0 and backend 4.1, this would be seen by the deploy scripts (and
operations team) as a single changeset for the environment. So if frontend3.1
needed backend4.1 to function, we would specify a independent rolling
deployment, so for each service thats being deployed (starting with the
backend), we would roll out server by server at a specified interval, and we
would wait for that server to return a healthy health check with the specified
version. If this fails the deploy is rolled back and notifications are sent.

^ with this working correctly, we have to have the understanding with the
application developers that we will never release a backend that is not
backwards compatible with any components that rely on this. (beyond a cutoff
version, which those do exist too)

In practice this works quite well, but is not magic, and definitely requires
some up front development of automation.

So, sorry for the long post, I think this is a long winded way of saying, my
opinion is that this belongs in a configuration management, as the logic
encapsulating "what version goes where" gets more complex over time, and so
some code, logic, decision making, and failsafes all have to be carefully
thought out.

~~~
perlgeek
Thanks again for your detailed answer.

Currently I think the simplest way forward for us is to check versions of
dependent services in the monitoring health check, so that at least the
operators have a simple way to figure out missing dependencies. All other
options (blue/green, rolling, ...) require too much infrastructure automation
that we don't yet have.

Though it still itches me to write some generic, remote dependency management
that's flexible enough to be used in different deployment scenarios and with
diverse tools. Maybe later :-)

------
olalonde
The closest I've found to something like this is docker-compose though I'm not
sure it can be used reliably on clusters.
[https://nixos.org/disnix/](https://nixos.org/disnix/) might be worth looking
into though I haven't personally used it.

~~~
perlgeek
Thanks, disnix looks really interesting. I'll do some reading (and maybe
experimenting).

