
Go += Package Versioning - bketelsen
https://research.swtch.com/vgo-intro
======
munificent
I'm going to comment mostly on the parts of the proposal that I think are
wrong, but don't take this to be an overall negative response. I'm excited to
see smart folks working on this, and package management is a really hard
problem. There are no silver bullets to code reuse.

Context for those who don't know: I along with Natalie Weizenbaum wrote
pub[1], the package manager used for Dart.

 _> Instead of concluding from Hyrum's law that semantic versioning is
impossible, I conclude that builds should be careful to use exactly the same
versions of each dependency that the author did, unless forced to do
otherwise. That is, builds should default to being as reproducible as
possible._

Right on. Another way to state this is: Changing the version of a dependency
should be an explicit user action, and not an implicit side effect of
installing dependencies.

    
    
        import "github.com/go-yaml/yaml/v2"
    

_> Creating v2.0.0, which in semantic versioning denotes a major break,
therefore creates a new package with a new import path, as required by import
compatibility. Because each major version has a different import path, a given
Go executable might contain one of each major version. This is expected and
desirable. It keeps programs building and allows parts of a very large program
to update from v1 to v2 independently._

It took me several readings to realize that you encode the major version
requirement _both_ in the import string and in the module requirements. The
former lets you have multiple copies of the "same" module in your app at
different major versions. The latter lets you express more precise version
requirements like "I need at least 2.3, not just 2.anything".

I think it's really going to confuse users to have the major version in both
places. What does it mean if I my code has:

    
    
        import "github.com/go-yaml/yaml/v2"
    

But my go.mod has:

    
    
        require (
          "github.com/go-yaml/yaml" v1.5.2
        )
    

I don't know if the goal of this is to avoid lockfiles, or to allow multiple
versions of the same package to co-exist, but I think it's going to end up a
confusing solution that doesn't cleanly solve any problem.

For what it's worth, Dart does not let you have two versions of the same
package in your application, even different major versions. This restriction
does cause real pain, but it doesn't appear to be insurmountable. Most of the
pain seems to be in the performance issues over-constrained dependencies
caused in our old version solver and not in the user's code itself.

In almost all cases, I think there is a single version of a given package that
would work in practice, and I think it's confusing for users to have an
application that has multiple versions of what they think of as the "same"
package inside it. This may be less of an issue in Go because it's
structurally typed, but in Dart you could get weird errors like "Expected a
Foo but got a Foo" because those "Foo"s are actually from different versions
of "foo". Requiring a single version avoids that.

 _> I believe this is the wrong default, for two important reasons. First, the
meaning of “newest allowed version” can change due to external events, namely
new versions being published. Maybe tonight someone will introduce a new
version of some dependency, and then tomorrow the same sequence of commands
you ran today would produce a different result._

No, I think newest (stable version) is the right default. Every package
manager in the world works this way and the odds that they _all_ got this
wrong are slim at this point.

At the point in time that the user is explicitly choosing to mess with their
dependencies, picking the current state of the art right then is likely what
the user wants. If I'm starting a brand new from scratch Ruby on Rails
application today, in 2017, there is no reason it should default to having me
use Rails 1.0 from 2005.

 _Every_ version of the package is new _to me_ because I'm changing my
dependencies right now. Might as well give me the version that gets me as up-
to-date as possible because once I start building on top of it, it gets
increasingly hard to change it. Encouraging me to build my app in terms of an
API that may already be quite out of date seems perverse.

 _> This proposal takes a different approach, which I call minimal version
selection. It defaults to using the oldest allowed version of every package
involved in the build. This decision does not change from today to tomorrow,
because no older version will be published._

I think this is confusing _older_ versions and _lower_. You could, I suppose,
build a package manager that forbids publishing a version number lower than
any previously published version of the package and thus declare this to be
true by fiat.

But, in practice, I don't think most package managers do this. In particular,
it's fairly common for a package to have multiple simultaneously supported
major or minor versions.

For example, Python supports both the 2.x and 3.x lines. 2.7 was released two
years after 3.0.

When a security issue is found in a package, it's common to see point releases
get released for older major/minor versions. So if foo has 1.1.0 and 1.2.0 out
today and a security bug that affects both is found, the maintainers will
likely release 1.1.1 and 1.2.1. This means 1.1.1 is released later than 1.2.0.

I think preferring minimum versions also has negative practical consequences.
Package maintainers have an easier job if most of their users are on similar,
recent versions of the package's own dependencies. It's no fun getting bug
reports from users who are using your code with ancient versions of its
dependencies. As a maintainer, you're spending most of your time ensuring your
code _still_ works with the _latest_ so have your users in a different
universe makes it harder to be in sync with them.

Look at, for example, how much more painful Android development is compared to
iOS because Android has such a longer tail of versions still in the wild that
app developers need to deal with.

If you do minimum version selection, my hunch is that package maintainers will
just constantly ship new versions of their packages that bump the minimum
dependencies to forcibly drag their users forword. Or they'll simply state
that they don't support older versions beyond some point in time even when the
package's own manifest states that it technically does.

There is a real fundamental tension here. Users — once they have their app
working — generally want stability and reproducibility. No surprises when they
aren't opting into them. But the maintainers of the packages those users rely
on want all of their users in the same bucket on the latest and greatest, not
smeared out over a long list of configurations to support.

A good package manager will balance those competing aims to foster a healthy
ecosystem, not just pick one or the other.

[1]: [https://pub.dartlang.org/](https://pub.dartlang.org/)

~~~
dkarl
_For what it 's worth, Dart does not let you have two versions of the same
package in your application, even different major versions. This restriction
does cause real pain, but it doesn't appear to be insurmountable. Most of the
pain seems to be in the performance issues over-constrained dependencies
caused in our old version solver and not in the user's code itself.

In almost all cases, I think there is a single version of a given package that
would work in practice, and I think it's confusing for users to have an
application that has multiple versions of what they think of as the "same"
package inside it. This may be less of an issue in Go because it's
structurally typed, but in Dart you could get weird errors like "Expected a
Foo but got a Foo" because those "Foo"s are actually from different versions
of "foo". Requiring a single version avoids that._

I think this makes a strong case for not releasing major version upgrades that
use the same package names. The very idea of two incompatible things having
the same name should set off alarm bells. Instead of trying to make that work,
we should be avoiding it.

In the absence of this principle, the Java ecosystem has developed a
compensatory mechanism of packaging "shadowed" versions of their dependencies
alongside their own code. This is an ugly hack to accomplish the same thing
after the fact, so we are already incurring even more complexity than would be
imposed by following this rule.

~~~
munificent
_> I think this makes a strong case for not releasing major version upgrades
that use the same package names. The very idea of two incompatible things
having the same name should set off alarm bells. Instead of trying to make
that work, we should be avoiding it._

If you do that, I think you'll find in practice that one of two things happens
(or more likely, _both_ , in a confusing mixture):

1\. People start releasing packages whose names include version numbers.
"markdown2", etc. Then you get _really_ confusing hallways conversations like,
"Yeah, you need to use markdown2 1.0.0."

2\. People start coming up with weird confusing names for the next major
version of packages because the current nice name is taken. Then you get
confusing conversations like, "Oh, yeah, you need to upgrade from
flippitywidget to spongiform. It's almost exactly the same, but they removed
that one deprecated method." Also don't forget to rename all of your imports.

~~~
dkarl
_I think you 'll find in practice that one of two things happens_

I think the existence of those practices (like Java dependency shading) proves
that people are struggling towards this solution on their own, without support
from the language or the community. With official support, if major versions
work the same way for everybody, it won't need to be so janky.

In practice, I predict that people would start behaving better, doing what
they should have been doing (and what many _have_ been doing) all along:
avoiding unnecessary breaking changes in non-0.x libraries, choosing function
signatures carefully, and growing by accretion and living with their mistakes.
Right now, I think some developers see major version bumps as a convenient way
to erase their mistakes, without taking into account the cost imposed on users
who end up juggling dependency conflicts.

------
Animats
I have misgivings about all this version pinning. At first, it seems to make
things easier. Programs don't break because some imported package changed. So
it looks like a win. At first.

Over time, though, version pinning builds up technical debt. You have software
locked to old unmaintained versions of packages. If you later try to bring
some package up to date, it can break the fragile lace of dependencies locked
in through past version pin decisions.

Some version pinning systems let you import multiple versions of the same
package into the same build to accommodate dependencies on different version.
That bloats executable code and may cause problems with multiple copies of the
same data.

On the other side, once package users are using version pinning, it's easier
for package maintainers to make changes that aren't backwards compatible. This
makes it harder for package users to upgrade to the new package and catch up.
Three years after you start version pinning, it's going to be a major effort
to clean up the mess.

The Go crowd tries to avoid churn. Their language and libraries don't change
too much. That's a good, practical decision. Go should reject version pinning
as incompatible with the goals of Go.

~~~
stickfigure
This is horrible advice.

I work in a lot of languages (including Go), which gives me some perspective.
On the extreme ends of this issue we have Maven, which specifies hard version
numbers (fuzzing is possible but culturally taboo) and NPM, which only
recently acquired the ability to lock version numbers and still has a strong
culture of "always grab latest".

The Maven approach is unquestionably superior; code always builds every time.
If you walk away from an NPM build for nine months, the chance that it will
build (let alone work as before) is vanishingly small. I waste stupid amounts
of time fighting with upstream dependencies rather than delivering business
value.

People break library interfaces, that's just a fact of life. The only question
is whether they will break your code when you explicitly decide to bump a
version, or when randomly when someone (possibly not even you) makes changes
to other parts of the system.

~~~
kodablah
> The Maven approach is unquestionably superior; code always builds every
> time.

If that's your goal. There's a middle ground which includes security updates:
asking the community to follow semver. Rust devs seem to do it well and my
crates build well with non-exact version numbers after not visiting them for a
while. Not sure why the majority of Rust devs do a good job with this and JS
devs don't. I suppose it's the strictness of the contract in the first place.

With Go, I've found popular library maintainers doing a good job at BC
contract compat even when there has been only one blessed version. I don't
assume giving them a bit more flexibility is going to hurt anything.

~~~
wtetzner
> There's a middle ground which includes security updates: asking the
> community to follow semver.

No, that doesn't work. People make mistakes, and you end up not being able to
build software. It might work _most_ of the time, but when things break, it's
of course always at the worst possible time.

I think Cargo got it right: use semver, but also have lock files, so you can
build with _exactly_ the same dependencies later.

~~~
woolvalley
With strong typing, you can analyze the code and automatically increment the
semver versions based on public API changes. Elm does this AFAIK. Reduces the
amount of mistakes that are possible.

~~~
dragonwriter
> With strong typing, you can analyze the code and automatically increment the
> semver versions based on public API changes.

This is nice, but should be noted does not absolutely prevent missing
backward-incompatible changes; that a function has the same signature does not
mean that it has backward compatible behavior.

(With a _rich enough_ strong, static type system, used well to start with, you
might hope that all meaningful behavior changes would manifest as type changes
as well, but I don't think its clear that that would actually be the case even
in a perfect world, and in any case, its unlikely to be the case in real use
even if it would be in the ideal case.)

------
readams
Damn it. Just when dep seemed like it was going to finally end the horrible
nightmare that is go dependency management, this comes along. There's this
great survey of all the progress that had been made and then "but we're doing
it in this bizarre terrible way because..."

------
baq
this sentence

> First, the meaning of “newest allowed version” can change due to external
> events, namely new versions being published.

is the root of what's wrong with this proposal - namely conflating time of
dependency resolution with the time of build. cargo (using it because it's
been referenced by the post) solves this problem by telling you to check in
the cargo.lock that it generates whenever you 'cargo update'. this means that
your version control preserves the build dependencies, just as it preserves
build recipes (e.g. build.rs or travis or whatever). in this world, there's no
'newest allowed version', because there is only one version.

wise men before me said 'as simple as possible, but no simpler' and I'm afraid
this proposal is too simple.

------
Sir_Cmpwn
>End of GOPATH

Yes yes yes yes finally! This has been the #1 reason I've avoided Go for years
and even my recent forays back into the language were only after I figured out
some hacks that I could use to avoid it.

~~~
warent
I've been programming in Go for a couple of years now and found that this need
to use "hacks ... to avoid it" is totally naive and irrational.

Agreed, GOPATH is awkward--at first. Because it's different from what most
programmers are used to. But once it's adopted it actually makes a lot of
sense, and looking back it seems bizarre to ever have such a strong desire to
avoid it at all costs--and the costs are high.

The same applies for Gofmt as well.

~~~
Sir_Cmpwn
I should add that another big part of the reason I've been avoiding the
language for years is the cult of Golang telling me my complaints are "totally
naive and irrational".

~~~
Groxx
This kind of attitude around package management and _especially_ dependency
injection have been major turn-offs for me as well. More than a sufficient
number of the community has been emitting a high-pressure stream of NIH
combined with ignorance of existing solutions, and it has been crippling the
community for years.

~~~
neoasterisk
Dependency injection is very commonly used in the Go ecosystem. Do not confuse
resistance about unidiomatic DI frameworks with resistance about DI itself.

~~~
Groxx
Which ones are common? I've been hoping this would be the case, but I've been
unable to find evidence for it.

e.g. the most-popular one I can see so far is Facebook's "inject", which has a
remarkably small 841 stars:
[https://github.com/facebookgo/inject](https://github.com/facebookgo/inject)

Even Glide, Godep, and Dep have several thousand. And those can barely claim
"most", much less "very common".

~~~
neoasterisk
What do you mean which ones? I am talking about dependency injection as a
technique/pattern. In Go you can just do it without importing anything.

Please read:

[https://medium.com/@benbjohnson/standard-package-
layout-7cdb...](https://medium.com/@benbjohnson/standard-package-
layout-7cdbc8391fc1#db5a)

(I've linked to the exact point in the article where it talks about DI but you
might want to read the rest as well.)

~~~
Groxx
Yeah, it's widespread as a hand-coded thing. It has to be - you essentially
have no choice in Go for even basic things like tests.

But that just means that everyone rolls their own thing by hand, with widely-
varying feature-sets, method names, bugs, etc. DI frameworks are very much
nicer for interop between a bunch of libs.

------
steveklabnik
Glad to see this! I'm still digesting the details, but to comment on
[https://research.swtch.com/cargo-
newest.html](https://research.swtch.com/cargo-newest.html)

Cargo does not use the latest version. Saying

    
    
      toml = "0.4.1"
    

is the same as saying

    
    
      toml = "^0.4.1"
    

NOT saying

    
    
      toml = "=0.4.1"
    

which is what rsc would guess.

This decision was made because ^ is the most reasonable default; over-use of
`=` destroy's Cargo's ability to help you upgrade, given that you basically
asked it to not think about it at all. Further, it would significantly bloat
binaries, as this would mean many more duplicate versions of packages in your
project. Basically, idiomatic use of = is only in very specific situations.

We could have required that you always specify some operator, but we also like
reasonable defaults. Writing `^` everywhere is just more typing.

The transitive dependency issue isn't any different, really: that's due to
said packages also not using =.

~~~
rsc
Hi. As I said in that page, I do know that cargo is working as designed, and
that "0.4.1" is the same as "^0.4.1". My point is maybe a little more subtle,
that cargo takes the newest allowed under the constraints, so given "^0.4.1",
it has a choice between 0.4.1, 0.4.2, 0.4.3, 0.4.4, and 0.4.5, and it takes
the last.

When you're adding a new direct dependency, taking the latest is almost
certainly right. But when you're pulling in a new transitive dependency, I
think taking the last is problematic: it seems much safer to me to take the
one closest to what the author of that code tested with.

I'll elaborate more on this in a post tomorrow.

For what it's worth, I think I do understand what cargo is doing, and why, and
I completely respect that approach. I just think this other approach might
have some different properties worth exploring. As I've been working on this
I've been using cargo as the gold standard, really. If the end of this all is
that people say Go's package management is as nice as cargo, then I will be
very very happy. A long way to go yet for sure.

~~~
Manishearth
I think your wording there is incorrect and very misleading, even if you do
understand it. You say

> Cargo selects the latest version of a new dependency and its own new
> dependencies

but that's not true; it selects the "latest semver-compatible minor version",
which is a pretty different thing. The way you've phrased it makes it seem
like cargo just flat out selects the newest version period (which can cause
breaking changes and a whole lot of pain).

So of course "people are frequently surprised", your actual statement is
incorrect and misleading. "Blindly updating to new versions" is a completely
different (and scary) proposition from "updating to semver compatible minor
versions". The latter still has its issues (I think the maximally minimal
approach that vgo is taking is a pretty neat solution to this and some other
issues) but it's a much more contained set of issues.

~~~
slrz
> but that's not true; it selects the "latest semver-compatible minor
> version", which is a pretty different thing.

No, it's exactly the same thing. Russ already established that an incompatible
module is not just a later version but, in fact, a totally different thing.

------
eximius
> Minimal Version Selection

Means no security fixes at the price of, well, minor developer inconvenience?
What is the inconvenience, exactly?

> ...tomorrow the same sequence of commands you ran today would produce a
> different result.

I mean, I guess this is technically true. But seems like it shouldn't be an
issue in practice as the API you're calling shouldn't have changed, just the
implementation. If it has changed, then downgrade the dependency?

~~~
binarycrusader
As a long-time maintainer of various packaging systems (and co-author of one):

I find myself wholly in agreement with the idea that maximal version selection
is appropriate for operating systems and general software components, but not
necessarily desirable for a build system.

When you consider the evaluation of a dependency graph in the context of a SAT
solver, you realize that the solver would consider both a minimal and maximal
version as potentially satisfying the general constraints found in a
dependency graph. Whether you then optimize the solution for a minimal or
maximal version becomes a matter of policy, not correctness.

The security concern can be appropriately addressed by increasing the minimum
version required in the appropriate places.

With all of that said, I think that Go's potential decision to optimize for
minimal version selection is likely to be considered a bug by many because it
will not be the "accepted" behavior that most are accustomed to. In
particular, I can already imagine a large number of developers adding a fix,
expecting others to automatically pick it up in the next build, and being
unpleasantly surprised when that doesn't happen.

This is an interesting experiment at the least, but I hope they make the
optimizing choice for minimal/maximal version configurable (although I doubt
they will).

~~~
munificent
_> When you consider the evaluation of a dependency graph in the context of a
SAT solver, you realize that the solver would consider both a minimal and
maximal version as potentially satisfying the general constraints found in a
dependency graph. Whether you then optimize the solution for a minimal or
maximal version becomes a matter of policy, not correctness._

I believe rsc is hoping they can avoid the need for a SAT solver entirely by
going with minimum versions (and, implied, not allowing a newer package to be
published with a lower version number).

~~~
uluyol
Actually, I think the SAT solver is avoided by making the only version
constraints of the form <=.

Using the min version appears to eschew the need for lock files. Want to
upgrade? Bump your min version.

~~~
munificent
_> Using the min version appears to eschew the need for lock files._

This only works if the system also prevents you from publishing a version of a
package with a lower number than any previously-published version of that
package.

So, for example, after you've shipped foo 1.3.0, if a critical security issue
is found in foo 1.2.0, you can't ship foo 1.2.1. Instead, all of your users
have to deal with revving all the way up to foo 1.3.1 where you're allow to
publish a fix.

It's not clear to me why they're trying so hard to avoid lockfiles. Lockfiles
are great.

~~~
joncalhoun
You _CAN_ ship v1.2.1 after v1.3.0 is live. I have tested this with the vgo
prototype and it works fine (see github.com/joncalhoun/vgo_main):

    
    
        $ vgo list -m -u
        MODULE                          VERSION                    LATEST
        github.com/joncalhoun/vgo_main  -                          -
        github.com/joncalhoun/vgo_demo  v1.0.1 (2018-02-20 18:26)  v1.1.0 (2018-02-20 18:25)
    

Notice that v1.0.1 was released _AFTER_ v1.1.0

What the minimum version is doing is giving our code a way to automatically
resolve upgrades if they are necessary. Eg if module X requires module Z w/ a
version >= 1.0.1, while module Y requires Z with a version >= 1.1.0 we clearly
CANNOt use v1.0.1, as it won't satisfy the requirements of Y, but we CAN use
v1.1.0 because it satisfies both.

The "minimum" stuff basically means that even if a version 1.3.2 of Z is
available, our code will still use v1.1.0 because this is the minimal version
to satisfy our needs. You can still upgrade Z with vgo, or if you upgrade a
module X and it now needs a newer version of Z vgo will automatically upgrade
in that case (but to the minimal version that X needs), but random upgrades to
new versions don't just occur between builds.

~~~
munificent
What happens if:

1\. I depend on foo with constraint ">1.5.0". The current minimum version of
foo that meets that is 1.7.0.

2\. Later, foo 1.6.0 is published.

3\. I run go get.

If I understand the proposal correctly, that go get will now spontaneously
downgrade me to foo 1.6.0. That defies the claim that builds are always
reproducible.

~~~
Groxx
So, I think you're right... but this is only a flaw if you as a user specify a
lower bound that does not exist. The tool won't do this. And it can be
prevented by disallowing referring to versions that don't exist.

It's entirely valid (and interesting! I hadn't thought of this one), but I'm
not sure if this would happen _even once_ IRL, except for people trying to
break the system. Which can be fun, but isn't _a risk_.

~~~
munificent
My experience from maintainer a package manager and trying to keep the
ecosystem healthy — which mirrors my experience on lots of other systems with
many users — is that anything your system allows people to do will be done at
some point.

~~~
Groxx
heh, good point :)

as always, of course there's a relevant xkcd:
[https://xkcd.com/1172/](https://xkcd.com/1172/)

------
xrstf
A lot of this seems to hinge on "go get" being the best way to get Go source
code. Most discussions I've read so far come down to "well, this might be
cool, but it would break go get, so we cannot ever do it".

I for one could live without go get alltogether. In 99% of cases it boils down
to a simple git clone anyway, which could (should?) be done by the package
manager anyway.

Maybe it's time to re-evaluate the existence of go get, now that we're seeing
that "just clone master and hope nothing breaks" has obviously not worked for
everyone outside of Google? Maybe bundling Go (the compiler and linker) with
tools for fetching source code wasn't such a good idea after all?

Just my 2 cents...

~~~
acdha
I find it bizarre how unnecessarily hard that’s been made to avoid a slight
amount of extra typing. The github imports work with the most basic options
but I’ve seen people waste a lot of time on internal servers, SSH vs. HTTPS,
tagging, etc. where it’d have been cleaner, easier and forward-compatible if
you just gave it a URL and it shelled out to run “git clone <URL>”.

------
howeyc
I like the removal of GOPATH, even though I don't mind it at all, it can be an
annoyance, especially when I don't want to co-mingle work/personal stuff. Now
I don't need to worry about setting GOPATH based on what I'm working on. Plus
sometimes I'd wonder, is it all in vendor, or is it using GOPATH for
something.

Plus now libraries are "modules" so libraries can have dependencies for
specific versions, before the question is what do you do if multiple
dependencies have the same dependencies in their own separate vendor
directories. This change removes that, as it's handled by the vgo tool for all
modules in that build.

go.mod is kind of a cross between lock-file and dependency listing. I think it
will work alright.

All together, it seems to be a cross between gb and dep, while also attempting
to solve library-packages tracking dependencies too.

------
gkya
Highly suggest to read the demo at [https://research.swtch.com/vgo-
tour](https://research.swtch.com/vgo-tour), which really clarifies everything.
I do not write Go, but I did write some bits in it in the past, and now think
that this will be a very useful addition to the Go toolchain. When I looked at
Go for the first time I really disliked the simple but sort-of castrated go-
build, and didn't really follow the developments since then, but this seems to
solve quite a bit of problems. I especially like how you can shadow a certain
dependency with a given package at a given path, and how it facilitates
vendoring. Hope this becomes included in go soon.

------
teacpde
If there is neither GOPATH nor vendor directories, where will the downloaded
packages be?

I digged vgo a bit and found the downloaded packages currently reside in
`$GOPATH/src/v/cache`, and seems like vgo won't work without GOPATH for now
([https://github.com/golang/vgo/blob/b6ca6ae975e2b066c002388a8...](https://github.com/golang/vgo/blob/b6ca6ae975e2b066c002388a896833851a9e54a9/vendor/cmd/go/internal/vgo/init.go#L111)).

------
shkkmo
I think that the choice to use "prefer minimum version by default" going to
confuse a lot of new developers used to the industry standard being the
opposite. If you are going against the industry default, there should be a
strong reasons. I don't think reasons provided really justify this.

> First, the meaning of “newest allowed version” can change due to external
> events, namely new versions being published. Maybe tonight someone will
> introduce a new version of some dependency, and then tomorrow the same
> sequence of commands you ran today would produce a different result.

This is why we have lock files and they work better for this.

While vgo's approach does allow reproducible builds, it doesn't allow you to
guarantee reproducibility the way that a lock file that has the specific
commit hashes does. With specific commit hashes you can verify the the version
you are building with in production is the exact same code as the one your
security team audited prior to release (assuming git has fixed its hash
collision vulernability).

You can get around this by abandoning version constraints in your go.mod file,
but then you have to track these version constraints out of band and manually
figure out what commit hash to stick in go.mod

You could also get around this by storing the hases for the approved versions
out of band and creating a build script that verifies these hashes prior to
building.

Both of these workarounds seem to defeat the point of having a standard
package manager in the first point.

> Second, to override this default, developers spend their time telling the
> package manager “no, don't use X,” and then the package manager spends its
> time searching for a way not to use X.

If you are concerned with allowing users to override this default, why not
have directive to override this default that can optionally be added to each
requirement in the go.mod file. This avoids an unexpected default and doesn't
force people who want the industry standard default to use a script to set
which dependencies use or don't use the -u flag with 'go get'.

~~~
Zamicol
>With specific commit hashes you can verify the the version you are building
with in production is the exact same code as the one your security team
audited prior to release (assuming git has fixed its hash collision
vulnerability)

Can we take a moment and point out that the hash collision vulnerability is
still at large with no ETA? This is after years of it being considered
insecure

I feel like this point continues to be glossed over with versioning systems
that depend on git commit hashes.

Bruce Schneier warned in February 2005 that SHA needed to be replaced.
([https://www.schneier.com/blog/archives/2005/02/cryptanalysis...](https://www.schneier.com/blog/archives/2005/02/cryptanalysis_o.html))
Git development didn't start until April 2005. So before git was even
developed, SHA was identified as needing to be deprecated.

So now, 13 years later, this is still an issue.

------
mCOLlSVIxp6c
I predict that under the proposed minimum version selection system some
package will decide it's important and flawless enough that users should
always depend on the most recent version. The package will recommend that
users depend on version v1.0.0 but the first release will be v1.100.0.
Subsequent releases will decrement the minor version to make sure newer
releases are selected.

~~~
overcyn
User's wouldn't be able to depend on version v1.0.0 if it doesn't exist. From
what I understand the tool isn't downloading the list of all versions from
GitHub and selecting the minimum. Its searching for the maximum version number
(i.e. minimal required version), from your go.mod file and all your
dependencies' go.mod file.

~~~
mCOLlSVIxp6c
I think you're correct. So my nasty little hack wouldn't work. That's probably
a good thing.

------
mfer
As I look this over a couple things really jump out at me...

1\. VCS tags are mutable. That's lock files store revision ids. Go is being
used to build immutable infrastructure but the proposed package management
system uses mutable versions.

2\. The proposal is less featureful that dep, npm/yarn in JS, composer in PHP,
maven, crates for rust, and others. I wonder how people will react to that.

~~~
Groxx
Mutable tags are my primary concern here, yeah. It seems pretty mitigate-able
by using that (wonderful IMO[1]) `v1.2.3-date-sha` syntax though - it's just
not human editable[2].

[1]: it fixes so many readability problems with SHA-pinned lock files, easily
shows downgrades in `diff` output, and `sort` likely produces the exact result
you wanted.

[2]: which may not be a problem, since you could in theory just re-run the
tool to fix it when you enter "v1.2.3" by hand.

------
readams
One that that I'm not able to tell from the proposal: is the /v1 at the end of
an import path a magical value that the tool handles, or do I literally need
to have that directory in my repo and have a version history represented at
the top level of every project?

Why not do the symbol mangling in the compiler if the goal is to have multiple
simultaneous versions built in? This is easy to make backward compatible since
it could default to v1.

------
icholy
Like all Go things, it looks kinda strange. But after playing around for a
bit, it feels really good. The minimal version selection simplicity is genius.

------
stock_toaster
I feel like the Go devs optimized implementation simplicity early on (likely a
sensible choice given Google's "one giant repo" workflow), and that has
resulted in some not-insignificant ecosystem complexity later (for others not
using "one giant repo" workflows).

It will be interesting to see how this plays out in the longer term.

------
marcrosoft
Please don't remove vendoring...

~~~
LaGrange
Yeah, this.

You want reproducible builds? Vendor your crap. Check it into your repository.
Live free.

~~~
hota_mazi
Or adopt a system with a sane versioning system and immutable artifacts, e.g.
Maven Central.

~~~
merb
which still means that your ci needs to download this stuff. with vendor it is
already checked it. consider builds on docker etc, where build caching is
hard. go is a breeze.

build takes like half a second because everything is already there. mvn will
first download the world. I'm a scala developer and sbt will probably take
like 10 minutes just downloading stuff, if a cache missed (even with a local
proxy).

~~~
pjmlp
I wonder how it is possible to take 10 minutes, unless it is downloading
directly from internet.

We always have internal Plexus mirrors, and the Jenkins servers have their
global .m2 local repository.

Even big JEE applications barely take more than 5 minutes to build.

The only build system I really dislike are Android builds with Gradle, trying
to beat C++ compilation times.

~~~
merb
well sbt uses ivy which is way slower than maven, when it comes to resolving
stuff.

------
mbertschler
Am I totally missing something in this document or does it not talk about
where these modules either as zips or source code are stored? It mentions
$GOPATH and vendor as not necessary anymore.

Where will the actual code be?

------
j_bond
I am curious why go doesn't just vendor dependencies like npm does when
confronted with conflicting requirements instead of trying to figure out a
version that satisfies all worlds? I always found this to be a nice feature
when working with npm.

I am also firm believer in version-pinning/lockfiles. Updating versions should
be a separate workflow with first-class built-in tools to support it. I think
that is the area where most package managers fall flat. They basically rely on
the developer to do all the heavy lifting.

~~~
TheDong
nodejs's "require", which pulls different versions depending on which module
calls it, is a cool trick.

Unfortunately, it also has a downside, and in go that downside would be
noticeable.

Let's take one easy example: a logging library. Let's say I pull in "logrus
v1.0.1" and one of my dependencies pulls in "logrus v1.0.2". In my "main"
function, I set logrus's default loglevel to debug
(logrus.SetLevel(logrus.DebugLevel)).

If go did the thing nodejs does (a different copy of logrus for my dependency
than my main), the "logrus.SetLevel" would only affect my package, not any of
my dependencies; they'd still log at a default level.

This would be true of other things; "func init" code would run once per
dependency that had that library, not just once, maps wouldn't be shared,
pools, etc.

This is a lot more memory usage, but it's also really surprising, especially
in the case of logging.

I definitely prefer having only one copy of a library in memory and having
package-level variables (like loglevel) work as expected.

If that ends up failing and I need two different versions, vendor+import-path
rewriting allows an escape hatch

These days, npm actually tries to minimize and flatten dependency versions as
much as possible to avoid the huge memory tax.

------
codedokode
> This proposal ... deprecates GOPATH in favor of a project-based workflow

That would be awesome. GOPATH was an awful idea that works only for
distribution package managers.

------
tetraodonpuffer
How would you specify dependencies on a commit/tag in a particular branch
(say, during development I always want to build with the HEAD of the
development branch for this module)? I know I could clone what I want
somewhere and use a replace directive inside the go.mod file, but I was
wondering if there's any way to do this using only vgo.

~~~
teacpde

      To name untagged commits, the pseudo-version v0.0.0-
      yyyymmddhhmmss-commit identifies a specific commit made on 
      the given date.

------
Keats
Will there be an official proxy or will open-source projects still break when
someone moves/deletes a repository?

~~~
icholy
That's what local caches are for.

~~~
Keats
Sure but me cloning the repository of someone else won't give me that person
cache. Without a central cache, it might work on that person computer but not
on mine if a repository moved.

------
teacpde
On one hand, as a developer, I like a Go builtin package management authored
and supported by the Go team; on the other hand, I feel bad for the open
source contributors who worked hard on third party package managers. Even dep,
the officially supported PDM, seems supposed to fade.

~~~
remus
> Even dep, the officially supported PDM, seems supposed to fade.

I beleve this was always the plan. The dep readme certainly makes it sound
that way at least:

> dep is a prototype dependency management tool for Go. It requires Go 1.8 or
> newer to compile. dep is safe for production use.

> dep is the official experiment, but not yet the official tool. Check out the
> Roadmap for more on what this means!

------
platz
> A build of a module by itself will always use the specific versions of
> required dependencies listed in the go.mod file. As part of a larger build,
> it will only use a newer version if something else in the build requires it.

This seems fraught

~~~
icholy
I think you need a noun to go with that adjective.

~~~
platz
With danger

------
013a
This is a well reasoned proposal. I've got a couple qualms.

Let's say you care about strict build reproducibility. You want some form of
manifest + lock file that defines exactly what versions of dependencies your
package expects. Great; more power to you.

The only reason you'd want that is if you aren't storing your dependencies
with your source code in source control. Otherwise, what's the point? If you
were vendoring and committing it, you already have strict reproducibility, and
you have the "supported versions" defined in the git repositories that come
alongside your dependencies.

So, adding this manifest+lock file allows you to get away with not
vendoring+committing. Awesome. You've gained reproducibility, but certainly
not _strict_ reproducibility.

Dependency package maintainers can rewrite git history. They can force push.
They can push breaking changes with minor versions. They can delete their
repositories. All of these things have already happened in NPM, either
maliciously or accidentally; why do we think they wouldn't happen with Go?

If you want strict reproducibility but are expecting it with just a
manifest+lock file, without dependency vendoring+committing, you're not
getting it. Full stop.

So, really, by adding a manifest+lock file, you're adding a "third level" of
reproducibility (#2 on this list).

1\. Low Reproducibility: Pull HEAD on build.

2\. Partial Reproducibility: Use tooling to pin to a git commitish, pull this
on build.

3a. Full Reproducibility (Easy+Dirty): Vendor+Commit all dependencies.

3b. Full Reproducibility (Hard+Clean): Mirror your dependencies into a new
self-controlled git repo.

I am struggling to think of a solid use case that would find real value in #2.
I have no doubt that people _think_ it would be valuable, but then the moment
the left-pad author deletes his repository, you're _going_ to second guess
yourself. You didn't want #2; you wanted #3 and settled for #2 because #3 was
too hard or dirty and you convinced yourself that adding tooling and version
numbers was keeping you safe. Because semver, right?

Moreover, this comes at the cost of complexity, which is strictly against Go's
core doctrine.

Moreover, a #2 solution looks strikingly similar to gopkg.in. Instead of
referencing Repository:HEAD we reference Repository:Commitish. The main
difference is that gopkg.in is an external service whereas this would be
controlled tooling. But by choosing #2 you're already exposing yourself to bad
actors, accidents, github going down, all of the above. So you're willing to
accept that exposure, but aren't willing to add that little extra exposure of
one more service?

I agree that the problem today, even with dep, isn't perfect. But I still
strongly believe that the answer lies somewhere in the ideas we already have,
not by importing ideas from NPM. We need a #3 that is Easy and Clean, not a
#2.

------
glasser
Is the "import compatibility rule" (ie, `/v2` suffixes in package paths)
implemented in vgo today?

------
hota_mazi
Amazing that in 2018, a language team needs to be convinced that versioning
packages is a good idea.

What's next, persuading them that generics are pretty cool too?

~~~
ovao
Considering there are existing and official dependency management tools for
Go, the Go team doesn't need to be convinced that versioning is a good idea.
This article is a discussion about mainlining a dependency management system
as part of the existing `go` tool.

They're similarly aware that generics are a desirable thing to have, but
haven't determined the ideal way to _implement_ generics into the language.

------
merb
this looks damn complex.

------
pcj128
Just use Bazel to build your go code. It already solved this problem.

------
leshow
How does Go not have package versioning? What do you guys use to get the
appropriate version of a dep?

~~~
abhia
There's a bunch of external packages. One by the go team:
[https://github.com/golang/dep](https://github.com/golang/dep)

~~~
iends
None of the top 10 of contributors to dep are from the Go core team, dep is a
community project that eventually clawed their way into official recognition
by sheer force of will by some great people.

------
pcj128
Just use Bazel to build your go code. It has already solved this problem.

------
pknopf
Generics first please, I can live with using tools like vndr and deep.

~~~
rsc
Sorry, maybe generics next.

~~~
pknopf
Thanks!

Vendoring needs to be solved and unified, I get it.

But to me, the biggest thorn is generics. I have ran into too many cases where
generics would have made my life a lot easier.

------
fiatjaf
> we also must not remove the best parts of the current go command: its
> simplicity, speed, and understandability.

Then you propose to introduce the concept of modules and make the world a
hell?

I believe versioning should be left of to third-party tools. For example, no
one is complaining about the lack of versioning built in Node.js.

