I have misgivings about all this version pinning. At first, it seems to make things easier. Programs don't break because some imported package changed. So it looks like a win. At first.
Over time, though, version pinning builds up technical debt. You have software locked to old unmaintained versions of packages. If you later try to bring some package up to date, it can break the fragile lace of dependencies locked in through past version pin decisions.
Some version pinning systems let you import multiple versions of the same package into the same build to accommodate dependencies on different version. That bloats executable code and may cause problems with multiple copies of the same data.
On the other side, once package users are using version pinning, it's easier for package maintainers to make changes that aren't backwards compatible. This makes it harder for package users to upgrade to the new package and catch up. Three years after you start version pinning, it's going to be a major effort to clean up the mess.
The Go crowd tries to avoid churn. Their language and libraries don't change too much. That's a good, practical decision. Go should reject version pinning as incompatible with the goals of Go.
I work in a lot of languages (including Go), which gives me some perspective. On the extreme ends of this issue we have Maven, which specifies hard version numbers (fuzzing is possible but culturally taboo) and NPM, which only recently acquired the ability to lock version numbers and still has a strong culture of "always grab latest".
The Maven approach is unquestionably superior; code always builds every time. If you walk away from an NPM build for nine months, the chance that it will build (let alone work as before) is vanishingly small. I waste stupid amounts of time fighting with upstream dependencies rather than delivering business value.
People break library interfaces, that's just a fact of life. The only question is whether they will break your code when you explicitly decide to bump a version, or when randomly when someone (possibly not even you) makes changes to other parts of the system.
> The Maven approach is unquestionably superior; code always builds every time.
If that's your goal. There's a middle ground which includes security updates: asking the community to follow semver. Rust devs seem to do it well and my crates build well with non-exact version numbers after not visiting them for a while. Not sure why the majority of Rust devs do a good job with this and JS devs don't. I suppose it's the strictness of the contract in the first place.
With Go, I've found popular library maintainers doing a good job at BC contract compat even when there has been only one blessed version. I don't assume giving them a bit more flexibility is going to hurt anything.
> There's a middle ground which includes security updates: asking the community to follow semver.
No, that doesn't work. People make mistakes, and you end up not being able to build software. It might work _most_ of the time, but when things break, it's of course always at the worst possible time.
I think Cargo got it right: use semver, but also have lock files, so you can build with _exactly_ the same dependencies later.
Sorry for the confusion, but that's what I meant. I was addressing the use-exact-versions-all-the-time argument. Maven goes too far in that it is taboo to do anything but use specific versions. Cargo, composer, etc do follow the proper approach. Maven w/ version numbers like "[1.2,1.3)" shouldn't be so taboo IMO. The "commit lock files for apps and don't for libs" is also practical.
With strong typing, you can analyze the code and automatically increment the semver versions based on public API changes. Elm does this AFAIK. Reduces the amount of mistakes that are possible.
> With strong typing, you can analyze the code and automatically increment the semver versions based on public API changes.
This is nice, but should be noted does not absolutely prevent missing backward-incompatible changes; that a function has the same signature does not mean that it has backward compatible behavior.
(With a rich enough strong, static type system, used well to start with, you might hope that all meaningful behavior changes would manifest as type changes as well, but I don't think its clear that that would actually be the case even in a perfect world, and in any case, its unlikely to be the case in real use even if it would be in the ideal case.)
Rust actually uses lock files, just not for libraries.
I've been bit by this a few times when I've come back to code 2-3 months later and forgot to include the .cargolock so I pretty much always use the lock file these days.
Yeah it's crazy that there's still so much controversy around this topic considering Node and Ruby have had amazing dependency management for over half a decade at this point. Dependency management in those languages is pretty much a solved problem and the fact that go isn't there yet drives me nuts since I have to work with it every day for my job.
Node and Ruby have had amazing dependency management for over half a decade at this point.
Eh?
NPM didn't have package-lock.json until v5, released in 2017. Before then there was the optional shrinkwrap that nobody used, so builds were totally unreproducible.
Ruby at least had Gemfile.lock from early days. Unfortunately there have been so many compatibility problems with different versions of Ruby itself that someone needed to invent rvm, rbenv, and chruby. Getting every dependency to behave in the same Ruby version was sometimes an odyssey. Still, at least builds are reproducible... as long as you're running on the same OS/CPU arch (uh oh native code!)
Ruby is actually pretty alright given the constraints, but Node/NPM is the canonical example of how NOT to do dependency management, and they're still trying to figure out here in 2018.
In my experience NPM shows exactly how to build a package manager. They are slowly fixing the problems one by one but half a decade ago NPM was terrible compared to what it is today.
Indeed this thread has caused me to retreat back into my Ruby hole. Seems like any contemporary solutions should be at least as good as a Gemfile/.lock.
The picture becomes more complicated when you consider a hierarchy of packages, because presumably the packages you depend on would have their own lock files, which represent the versions those packages have been tested with.
npm will pick the latest version of the dependencies compatible with the package.json of the packages (package-lock.json, is not published as part of the package). This means that with npm we may end up using a version of a transitive dependency which is much later than than the one that your direct dependency was tested with.
The proposal for vgo will take the minimum version compatible with the packages, and hence pick a version which is closer to the actually tested one.
This proposed “minimum version” behavior will effectively prevent “automated” security updates, which is what most reasonable people expect from their package manager.
Consider all tens of thousands of CVE bugs found in widely used image, video, XML, etc. libraries. Even “updated” software is often compromised via outdated libraries.
With a “min version” approach, none of those will get patched without explicit action by a developer, who won’t do that because he doesn’t even know about the bug unless he reads the thousands of messages each day on bugtrak.
If anything, history has shown that we should be focusing package management on getting security fixes to the the end user as soon as possible.
Note that even currently, lockfiles have the effect of locking you to an old version of the software. You could choose to automatically update all dependencies to latest in CI, but then by that token you can run vgo get -u, to get the latest versions here as well.
One of the issues which the vgo developers point out is that the "latest" behaviour has the perverse effect that a module A may declare a dependency of version 1.1 of a module B, and may have never been even tested on that version because the package manager always picked the latest.
In some sense, the vgo approach is saying that across hierarchy of module owners, each one should keep upgrading their manifest to the latest versions of their direct dependencies, and ensure that things keep working, rather than relying on the topmost module to pull in the latest of the entire tree of dependencies. This seems to be good for the entire ecosystem.
> each one should keep upgrading their manifest to the latest versions of their direct dependencies, and ensure that things keep working
But that’s the problem. You’re relying on N people, including the package maintainers and end developers to all take timely action in order to get a security fix to the end user.
That simply won’t happen.
What should happen is a single package maintainer fixes a vulnerability, and that fix automatically flows through the system to all places where that package is used. And insecure versions should be made unavailable or throw a critical “I won’t build” error.
Perhaps some way of marking versions as security critical might help, but the proposed approach will leave tons of vulnerable libraries in the wild.
All the current package managers for other languages have this issue to some degree. Golang should do better with knowledge of those mistakes.
PHP package manager, Composer, uses lock files only for applications, not for libraries. The developer makes sure that everything works and then commits the lock file.
> People break library interfaces, that's just a fact of life.
Sure, but in my experience it happens at least 10x as frequently in NPM than with Go. It's really common for me to update all my Go dependencies and everything just continues to work. With NPM I have to start looking for migration documentation and change a bunch of my code.
For me, it was difficult that I couldn't have reproducible builds. I would kick off a deploy and it would one day break due to some dependency (incorrectly) making a breaking change. It's important to keep packages up to date, but I believe it should be a conscious decision of the developer.
I also think reproducable builds are very important. Vendoring has solved this for me in Go even before it was officially supported.
My main fear is that when people get used to the idea that their master repo doesn't have to have a backwards compatible interface, then updating dependencies will look awfully similar to NPM where authors change interfaces based on their weekly mood.
Maven doesn't specify hard version numbers - the default meaning of "1.0" is >= 1.0, and some packages specify dependencies without a version at all. It's quite possible to get updates of transitive dependencies without realising it (especially since most tooling does not encourage pinning any dependency other than the superficial ones).
We do reject version pinning. There is no way to pin a particular version. The only constraint you can express is ">= this specific version". If nothing else pushes it forward, though, you'll keep getting that specific version. But it's not pinning, it's just stating a minimum, and the system uses the single oldest version that satisfies all the stated minimums (the max of the mins).
That seems like it only works well if library authors are both extremely well behaved and also don't make mistakes, and there's no way that will be universally true. Every time I've had to pin a version it's because a backwards incompatibility issue or more likely just a bug is, intentionally or not, added to a library.
It's going to be much more painful if people have to fork when a bug is introduced lest they have to wait a week or more to get a bugfix upstreamed.
I think you misunderstood what was said. In this proposal, each of your modules specifies a minimum version, but when code is built it tries to use the minimal version that meets those requirements.
Eg I might have my module that says:
"some/pkg" v1.4.1
Which means I need at least version 1.4.1.
When the code builds it will try to use 1.4.1 EVEN IF 1.4.2 EXISTS, unless it is forced to use 1.4.2 by another module you depend on. That is, say you are using module X and it says:
"some/pkg" v1.4.2
At this point 1.4.1 cannot be used - module x, which you are using, won't work with 1.4.1 - so pinning doesn't help. Your code will not build unless you manually update your pinned version.
What the minimal version selection does is say "okay one module needs >=1.4.1, another needs >=1.4.2. What is the minimal version that satisfies these requirements?" And the answer to that is `1.4.2`, so even if 1.4.8 exists, 1.4.2 is used in that scenario.
I don't know how this will work long term. I think there are definitely some concerns to consider (eg many minor version bumps are for security fixes), but the scenario you are describing - the newer version causing issues - just isn't an issue as I understand it because your code won't opt to use a newer version unless you (or another module you use) explicitly tell it to.
Gotcha, so it always chooses the minimum possible version, I was conflating multiple pieces from this thread. That would solve this possible issue at least, you're correct.
It sounds like you won't see the bug unless you commit a change to your own project's go.mod to change a version number. Assuming you test your changes before committing, your repo can never be broken by someone's commit to a different repo.
On the other hand, whenever you edit a direct dependency's minimum version, it could upgrade any other dependency via a cascade of minimum version upgrades. (In which case, don't commit it.) This keeps your team moving, but it might make dependency upgrades harder.
For example, it's up to each repo's owners (for applications, at least) to notice when there's a security patch to one of its transitive dependencies, and generate and test a commit. Someone will need to write a tool to do this automatically.
If the test fails: now what? I guess that's when you need to look into forking.
You can specify that a given dependency be replaced by another one. That only applies at the top level though, not when the go.mod file with the replace clause is used by another module.
The ability to depend on library versions that do not exist is a misfeature. It should not be possible for someone to build a new version of their software and cause your software to cease building or running.
This doesn't just result in non-reproducible builds, but it results in them at unpredictable times and, if you have servicing branches of your code, backward through time. This is not a good property if you need to know what you are building today is the same as what you built yesterday, modulo intentional changes, or even that it will build or run.
It is difficult to live without versions pinning in commercial projects. Imagine if you get an urgent task that needs to be completed today, but then you find out that the dependencies were updated, your project doesn't build anymore and you need to fix that first.
> Over time, though, version pinning builds up technical debt.
Version pinning does not. Misusing it might, but don't do that.
Version pinning of dependencies is a tool for assuring the behavior of releases, but in development you should be generally be updating to the latest stable version of dependencies and then pinning them for the next release, not keeping the old pinned versions.
Isn't technical debt usually just the situation of having many things that should be done, but aren't? In other words, the accumulation of "don't do that" cases in code.
> Isn't technical debt usually just the situation of having many things that should be done, but aren't?
Sure, the things that you shouldn't do are the things that lead to technical debt. What I'm saying is that "version pinning to assure consistent behavior in stable releases" is not one of those things.
What is one of those things is "leaving pins from the last release in place in development".
Version pinning in releases is a means of providing stability for downstream users. It should be encouraged.
Leaving those pins in place while developing the next version is a source of technical debt. It should be discouraged.
Technical debt is definitely something to worry about, however I think most companies' first priority is a working product.
Technical debt is to be handled during the development cycle, not in source code and most of the issues you bring up (bloated software, version pinning) are solved in build systems and again should not be handled by source code.
Anyone working in an enterprise enviorment, especially that has clients on different systems, would instantly crumble without being able to target specfic builds.
Legacy and long term support software is everything.
>Go should reject version pinning as incompatible with the goals of Go.
This would instantly make go a 'no-go' in any enterprise enviorment.
Don't think of it as /version/ pinning. Think of it as Interface / API pinning.
A //public library// should have a well thought out interface that works at least for one project and preferably for several different projects.
If the interface needs to change I agree with the recommended semantics of planning what that change should look like and creating a new named interface (API) for that specification.
Since go expects everything to build against the 'latest' version of an interface, it should be obvious what's wrong if a stale cached version is in use. Thus /expanding/ an interface / API is permissible. They should be thought of as /write only/.
Being able to create consistent builds seems more important to me than being forced to fix builds because random packages changed their API.
You're right that it is technical debt to not be on the latest version, but if I need to roll back to a previous (of my application) version due to a regression, I want to be 100% sure I get the exact same state as I did before. If my dependency management software doesn't solve this, it's not good enough.
Just because you don't need reproducible builds doesn't mean nobody needs them.
As soon as some tool is used to provide reproducible builds, you have effective version pinning.
If your organization values avoiding technical debt accumulation over reproducable builds, they don't don't have to use version pinning and can always run 'go get' with the '-u' flag.
> once package users are using version pinning, it's easier for package maintainers to make changes that aren't backwards compatible.
I don't think ease of making backwards incompatible changes with vgo. Both before and after vgo, backwards incompatible changes must go in a new major version / new import path. The ability to pin major versions already exists.
> Over time, though, version pinning builds up technical debt. You have software locked to old unmaintained versions of packages.
If you are talking about open source, it is not a problem: anyone can upgrade dependencies and make a pull request. Without version pinning, the project will always be in a "broken" state.
For example, recently I tried to run Firefox 3.5 on linux and it crashed with modern versions of gobject libraries although they are supposed to be backwards-compatible. I wish Debian package manager allowed to install an old version of a library specially for this application.
I am pretty sure this is exactly the problem semantic versioning is trying to solve. You only pin to the major version, which is supposed to ensure no breaking change occurs.
personally this is the main reason I have avoided go from the beginning. without being able to specify a version of the library I want to use, whose to say that the next time I try to run my program it might use a newer version of a library and not run. If that happens there was no easy way around it other than digging into the code and fixing the problem.
Since Go is compiled and statically linked there's no way that library version can change between runs of your program. It can change between compilations of your program though, the standard go solution to that is vendoring code which will guarantee that it's always the same code getting compiled. This is not without its issues, but I find it to be a pretty usable system, just smart enough to get what I need done. To be fair though, vendoring didn't exist in the beginning of go, so this was a perfectly good reason to avoid it then.
Over time, though, version pinning builds up technical debt. You have software locked to old unmaintained versions of packages. If you later try to bring some package up to date, it can break the fragile lace of dependencies locked in through past version pin decisions.
Some version pinning systems let you import multiple versions of the same package into the same build to accommodate dependencies on different version. That bloats executable code and may cause problems with multiple copies of the same data.
On the other side, once package users are using version pinning, it's easier for package maintainers to make changes that aren't backwards compatible. This makes it harder for package users to upgrade to the new package and catch up. Three years after you start version pinning, it's going to be a major effort to clean up the mess.
The Go crowd tries to avoid churn. Their language and libraries don't change too much. That's a good, practical decision. Go should reject version pinning as incompatible with the goals of Go.