
Dependency Drift: A Metric for Software Aging - colinbartlett
https://nimbleindustries.io/2020/01/31/dependency-drift-a-metric-for-software-aging/
======
jacques_chester
At Pivotal (now part of VMware, but it's easier to explain this way) we have
an increasing number of teams that track "vulnerability budgets" and "legacy
budgets", by way of analogy to "error budgets" from SRE approaches.[0]

Essentially, a vulnerability budget tracks days since last release for a
dependency or system; a legacy budget tracks time left until a dependency or
system falls out of support. Our CloudOps team and telemetry teams now plot
these for their own systems and customer systems, producing a characteristic
sawtooth pattern.

I think these are good steps forward, but I see some possible evolutions.

One evolution is to combine these into a more general risk framework like
FAIR, so that we're able to use a _single_ budget for all these categories of
risk.

Another is to consider risk exposure as a volume. At any given instant I have
a wide range of risks in play, distributed over a wide range of systems. As
time evolves these risks come and go. A single score (like the one sketched on
the far side of the link) only gives me one axis of risk. I also need to know
what systems are currently expressing that risk by the presence or absence of
software, giving me a second axis. Finally, these evolve over time, giving a
third axis.

The sum of volume for any time slice is what ought to be budgeted.

Another thing is to make these problems behave less like "big-bang control",
where we switch modes drastically when crossing a threshold value. What would
be preferred is a greater degree of smoothness.

And finally, to make all of this tractable: automation. All the metrics and
scorecards and reports and graphs in the world won't make this problem
_easier_. They can at best make it _visible_ and _demoralising_.

[0] This presentation by David Laing and Jim Thomson is a good introduction:
[https://www.youtube.com/watch?v=lXPmh9Ap114](https://www.youtube.com/watch?v=lXPmh9Ap114)

David & Tim are responsible for an ongoing multi-year systematic investigation
of upgrades as a technical and organisational behaviour.

------
guitarbill
The whole process of upgrading dependencies is annoying, mainly because stuff
keeps breaking. I currently try to keep everything up-to-date, simply because
then the number of breakages are spread out and easier to deal with, and when
critical fixes come through, they're easier to apply. Automation does help
with this. But ideally, I'd like the "it it ain't broke don't fix it"
approach. It feels like such a waste of time.

I would love to see Dependabot go further and track upgrade breakage. Being
notified about vulnerabilities is a great service and takes the load off. But
too many times, it still breaks.

Recent example: Checkstyle security issue, Dependabot cuts a PR. Great. Except
almost every minor release [0] has a breaking change. Some libraries are much
better than this, and in fact I ended up simply ditching Checkstyle. (It's
open source, they don't owe me anything. I get it.)

Until we quantify the breakage issue, you don't have enough data to make a
decision if upgrading dependencies is worth it outside of a security fix. And
in those cases, it's hard to make an argument keeping up to date is the best
choice (other than personal philosophy).

[0]
[https://checkstyle.org/releasenotes.html](https://checkstyle.org/releasenotes.html)

~~~
colinbartlett
I believe Dependabot quantifies this to some extent with a "Compatibility
score": [https://dependabot.com/#how-it-works](https://dependabot.com/#how-it-
works)

But it can only measure what people have test coverage for, presumably.

~~~
guitarbill
I did see that, but it's pretty opaque (some percentage). And when I click
through the badge, it takes me to GitHub's help page [0]. What would be
helpful is to link to other public PRs so you might be able to find out how
other people solved the issues. Is there any way to see historic compatibility
scores to evaluate if it's worth to drop a certain dependency?

Edit: For normal dependency updates, compatibility probably more useful. For
security fixes, what choice to you have other than to apply it, or spend more
time investigating and trying to come up with your own mitigation?

[0] [https://help.github.com/articles/configuring-automated-
secur...](https://help.github.com/articles/configuring-automated-security-
fixes)

------
yourpalkeith
A variant of this idea that would be more useful to me personally would be
some measure of how actively maintained my dependencies are. It would be nice
to know if my long-lived project depends on something that has been abandoned.

Details might be tricky, given that some very useful packages simply don't
need frequent updates.

~~~
k__
I like the drift metric more, because some libraries are simply stable and
don't need much updates anymore, this doesn't mean they are bad for your
system.

------
alangibson
I think you're on to something here. This is hot topic amongst our teams right
now. Dependency maintenance looks to me to be one of those things like code
duplication that correlates strongly with a multitude of sins.

It's also interesting to see someone else building a tool around a new metric.
I'm doing something similar (shameless plug -> [https://without.fail/estimate-
accuracy](https://without.fail/estimate-accuracy)) for determining how well
estimates correlate with actual development times. I could definitely take
some lessons from you because Dependency Drift is way more brandable than
Estimate Accuracy.

------
SideburnsOfDoom
Doesn't this just re-invent the Libyears metric?

[https://libyear.com/](https://libyear.com/)

I have skimmed through this article, it seems fairly long but light on
details. How does this differ from libyears? What actually is your metric?

That site (libyear.com) IMHO makes the concept refreshingly clear, this
"Dependency Drift" site seems vague and markety. It's trying to make a big
deal out of something, but what remains unclear after reading it. The libyear
site is the opposite - clear and not a big deal.

------
desc
The problem is that this is a bunch of bollocks. But at the same time, that
might actually resolve my problem with it as the pain gets fed back.

If one is _very very lucky_ , a library will have accurate changenotes
explaining what the version increment actually means, distinguishing between
security updates and wanking-over-new-subdependency-for-shiny-irrelevant-
features updates.

However, if people are penalised for not wasting their time chasing the latest
version of leftpad-with-custom-characters-except-in-locations-which-are-prime-
or-square.0.4.3-beta.butnotreally, maybe we'll see shallower dependency trees
in the important stuff.

Where 'important' ends up being defined as 'the packages which everyone else
gravitates to, and therefore can't be avoided'.

Ideally we'd see security updates for previous major versions of things, for
those of us without feature addiction, but that would demand more of the devs
producing this crap.

------
jefffoster
I agree this is an important metric and a kind of technical debt that's hard
to capture.

How do you discriminate this from RenovateBot or Dependa-Bot? Both of those
will not only track the drift but they'll also generate the PR to fix it.

~~~
colinbartlett
Co-author here, thanks for the input. We use Dependabot on all our projects
and love it. I've not seen any kind of numeric metric in Dependabot or any of
their competitors.

What I really want is a graph that shows, numerically, the drift over time.
You can show this to your stakeholders: "See, look! We haven't dedicated time
to stack upgrades and now look how far out of date we are creeping." And then,
consequently, I want to see that chart burn down to zero as you do stay on top
of things with the help of Dependabot or similar.

~~~
SideburnsOfDoom
This is the libyears metric ( [https://libyear.com/](https://libyear.com/) )
tracked over time.

Or are you using a different metric that you haven't defined for us?

