
An Analysis of vgo - bpineau
https://sdboyer.io/vgo/intro/
======
pcx
I think the current approach of announcing the death of `dep` before even
having a stable alternative (`vgo`) is deeply flawed. With `dep` the ecosystem
finally had something most of the developers agreed on, despite whatever
shortcomings it had. The model was working. They could have adopted dep and
improved on it. The packaging story for Golang has gone from worse to OK to
worse again.

After all this progress, we still don't have a stable alternative and no clear
date as to when it might be ready or even worse, the community is still not
sure vgo will be the final one. Golang should be about pragmatic, practical
decisions. The packaging situation is just the opposite.

~~~
lobster_johnson
I don't disagree — dep has been a huge improvement on Go's package management
story. Until dep was stable, we were using Glide, which is extremely buggy.

That said, if you ignore the doubts and arguments about the sanity of its
dependency resolution algorithm, I think vgo's introduction of modules is its
real contribution.

Go packages have been problematic since the start since their design, naively,
conflates a bunch of concepts that most developers keep separate: File
location, file structure and source repository. That is, when you do 'import
"github.com/foo/bar"', there's a whole bunch of conventions at play that
dictate where it can be fetched from and where it should be in the file
system. Aside from the fact that the $GOPATH convention is maddeningly
annoying to many developers, the design has several issues, such as that an
entire git repository has to be fetched in order to use a nested package, or
that a package can have multiple import paths. But getting rid of $GOPATH is
in itself a huge deal.

I'm personally less concerned with the new MVS algorithm (though I personally
prefer the more traditional lock file approach such as that used by dep,
Cargo, Bundler, NPM, etc.), as long as I can get modules. While getting
modules retrofitted into depwithout all the other stuff in the proposal would
be lovely, I don't know if the module design is too dependent (heh) on the
rest of vgo's versioning thesis to be separated out.

~~~
endymi0n
All of this feels like a classic case of "the perfect is the enemy of the
good". Package and dependency management in Go is honestly a total mess today
and with vgo challenging all current assumptions about how to generally
approach packagement, it brings complete paralysis to the simple question of
"how should I organize my dependencies _right now_ ".

As much as I loathe JavaScript or Ruby these days for their language, using
npm or bundler simply feels sane despite all of their warts. Yes, `left-pad`.
Yes, they do have their issues. But seriously, they just work and all open
source packages comply to their specifications. And they can and do evolve
gradually, like yarn shows.

It's just so funny how careful the Go community seems to tread about not
breaking their language with all these Go 2 proposals and general resistance
to new features — and then completely ignore this spirit to "move fast and
break things" when it comes to package management.

~~~
pknopf
> using npm or bundler simply feels sane

I feel the opposite. Lock files are awesome when your developing a package,
but after you release it, you are at the mercy of all your dependencies in
making sure that they don't break semvar. Each "npm install" will ignore any
lock file your package was using during development.

It would be awesome if I could distribute a package-json.lock file with my npm
package and have yarn/npm use it when resolving packages on fresh
installations.

Yeah, this would mean that there may be duplicated libraries in your
node_modules tree, each with a slightly different version, even if semver
compat. However, people break semver to often anyway.

I want to deploy a package to npm and be sure that it will _never_ break,
_ever_.

~~~
regularfry
Can't you do this right now by specifying exact dependency versions in
package.json? Lock files should be for applications, not libraries.

~~~
pknopf
Yes, but "npm install" will by default add a semver-compat package version.
The vast majority of packages out there use what "npm install" gives you be
default.

Furthermore, even if I use exact versions in my package.json, that doesn't
stop my referenced packages from internally referencing semver-compat
versions.

At the very least, you should reference exact versions at the top level. It is
better than nothing.

------
nicpottier
Though this is well written and I'm sure Sam is trying to set a tone of
respect and appreciation of what is good in vgo so as to start on the right
foot, he is also "burying the lead" in that he never gets to the point.

I think Russ did an unusually good job of explaining the ideas around vgo,
grounded by using real world examples. He set a high bar there, but hopefully
Sam adopts some of the same methods of very concrete examples to explain his
criticisms.

As it stands this is an interesting piece of reading but is too high level and
abstract in its criticism to be effective. Looking forward to future
installments that hopefully that will make Sam's criticisms clear.

From what I can tell one piece of criticism is the inability of vgo to define
incompatibilities. That is that if you know these versions won't work, you can
say never allow that version in your dependency graph. I'd love to hear more
examples of this in the real world. Since vgo dependencies are basically
always "pinned" this seems like that not big a deal to me, but maybe I'm
missing something. Perhaps this is something that occurs with having two of
your dependencies having different dependencies on a third package and one of
those works only with dep3@v1.2 and the other only works with dep3@v1.3 or
somesuch.

Maybe there are ways around that or maybe that's not the actual problem
(again, hope Sam explains more in the future). I think there is a really
interesting element to MSV that will actually drive Go culture though (and I'm
a big believer in the importance of the culture of a language) which I think
may minimize the problems that Sam is talking about and for the betterment of
the community.

As others have noted, I hope that Sam comes out with the rest of this series
quickly so that the go community can commit do a direction sooner rather than
later and move on to actually building this new ecosystem.

~~~
4ad
> he is also "burying the lead"

The expression is burying the lede, not lead.

~~~
nicpottier
Hah, thanks, learn something every day! [https://www.merriam-
webster.com/words-at-play/bury-the-lede-...](https://www.merriam-
webster.com/words-at-play/bury-the-lede-versus-lead)

------
pcwalton
The basic upshot of all this is that vgo's attempt to avoid NP-completeness in
the dependency resolution algorithm is solving a problem that doesn't matter
in practice, and is simultaneously creating downsides that _do_ matter in
practice.

This is consistent with what I've seen with systems like Cargo. If I had to
make a list of top 10 issues I run into with Cargo, theoretical scalability of
the core dependency resolution algorithm wouldn't make the cut.

~~~
enqk
On the other hand, git became successful by being fast and by not solving
certain of the hard theoretical problems that specialists were obsessed with
solving. It's hard to know before hand what the healthy compromise might be.

~~~
TheDong
This rings as a wrong interpretation of history.

When git became successful, there were scant few other options with a
comparable feature-set regardless of speed.

Its distributed nature, ability to easily merge and rebase sets of changes,
etc, were all wonderful solutions to real problems.

I'm unconvinced that its success was because it solved fewer problems than the
state of the art, but rather that it solved more. This is a stark contrast to
vgo which is intentionally solving fewer problems than other modern dependency
management tools.

In addition, git must be performant enough to handle a git repo (namely the
linux kernel, its original usecase). Beyond that, I think there's no evidence
it intentionally cut features or complexity to be faster.

In the case of vgo, it cuts additional user-facing features in order to be
faster, but there's 0 evidence that the speed matters, that there are any go
projects in the world that cannot be solved quickly enough with an
approximation of a sat solver.

~~~
zaphar
Mercurial was pretty close to contemporaneus. And when git was first launched
it was extremely bare bones and even more incredibly hard to use than it is
now. Darcs, Monotone, and Arch were slower and I think predated git. Darcs in
particular blew the doors off of git as far as ease of use given the problem
domain.

Git won almost entirely due to two factors.

1\. The linux kernel used it so that brought some prestige.

2\. Github made git hosting easy and free for a lot of people and had the
cultural cachet to drive adoption.

Git has a very solid underpinning. But it's user interface and lack of guard
rails has always made it painful. People endure the pain because if they don't
they won't be able to use the tool that their industry has chosen. But some of
us wish a different choice had been made in the dvcs arena.

~~~
deong
It's also important to remember just how far Git was ahead of most competitors
in performance. I was using monotone quite a bit around this time, and while
its UI was frankly not much better than git's, it at least worked acceptably
quickly. Darcs was mostly defined in the public eye by the exponential merge
issue. Hg was fine, for some definition of fine, but it was enough slower than
git to be annoying -- maybe not deal-breaking, but noticeable on pretty much
any task.

I remember installing cogito to act as a front-end for git because very early
on it seemed obvious that (a) git was going to win, and (b) it was going to
win in spite of its UI, which was saying something.

------
red_admiral
> If there are two algorithms that satisfy the same requirements, and only one
> is NP-complete, you pick the other one.

Problems can be NP-complete. Algorithms cannot - they can be exponential time
(and if your choice is between exponential and double exponential, you pick
the former!)

In practice, and especially for SAT solving, as far as I understand the state
of the art is that the general case is NP-hard but we have tools that work
surprisingly well in those cases that reality tends to produce. In some more
academic cases, we even have heuristic solvers that are not even guaranteed to
terminate, but usually do so within a couple of seconds, so you can get away
with the "beginner's solution to the halting problem".

~~~
munificent
_> In practice, and especially for SAT solving, as far as I understand the
state of the art is that the general case is NP-hard but we have tools that
work surprisingly well in those cases that reality tends to produce._

You're correct. Modern version solvers need _fairly_ sophisticated algorithms,
but it's not rocket science. It's mostly a non-problem for most users of most
package managers most of the time.

~~~
bennofs
The thing that worries me about this line of thinking is that as far as I
know, we don't know much about why SAT solvers tend to work well in practice.
So if I ever happen to hit a problem that the solver cannot solve, all we can
say is "well bad luck". If the algorithm fails, I won't have any way to know
why. This strikes me as a bad situation to be in.

------
jstarks
Could anyone distill this down for the casual reader? The author has a fairly
laborious style and seems to be writing for an audience that is intimately
familiar with the internals of dep and vgo.

~~~
lobster_johnson
The article is about vgo, a proposal (with working prototype) to build package
management into Go itself.

vgo bundles several improvements, a major one being the introduction of
"modules", which are versioned collections of packages. Modules, among other
features, will let us decouple the import path from the file system structure
and source repository, making the awkward and much-maligned $GOPATH finally
obsolete.

vgo also introduces a new dependency resolution algorithm, minimal version
selection (MVS), which is different from traditional dependency resolution
algorithms in that it will at any time choose the _oldest_ allowed version
that satisfy the minimum version constraints specified in the list of imported
packages, where other systems will choose the _newest_. As MVS does not
require (as with dep) a complex boolean SAT solver with potentially
pathological, NP-complete cases, it is much simpler and faster to compute.
However, it also has downsides. (Edited, thanks to pa7ch for correction.)

The author likes pretty much everything about vgo except the MVS algorithm,
which has been deeply controversial since the proposal was first published.
The author goes into explanations of why he thinks MVS is a bad fit, but this
is a complicated topic, so you really have read both the vgo proposal and this
rebuttal to understand what it's all about.

The article was written by Sam Boyer, one of the designers/authors of dep.
While he has been courteous about it, there are obviously emotions at play;
during its development, dep seemed to have the blessing of the official Go
team, and this proposal came out of the blue and probably felt like an ambush.

~~~
pa7ch
Minor correction: MVS requires modules to specify the minimum version to use
of a module. The algorithm will then select the _newest_ among those based on
the idea that semver is forward compatible among minor versions. The name
minimal version selection throws people off.

Other algorithms would select the newest release within a major version even
if no module has ever tested it and it was released 5 seconds ago.

When authors do make incompatible changes and MVS selects a broken dependency,
as expected, there are manual escape hatches. More complicated NP complete
algo's would have more of an automated answer here by allowing dependencies to
have boolean constraints (greater then X, less then Y, not Z etc.) on version
so that there is this collaborative summation of constraints to work around
authors violating semver or bugs. To summerize poorly: It seems Sam Boyer
regrets that dep didn't do what vgo does by default, but believes that a SAT
solver could kinda do MVS for the happy path but still solve against the
introduction of boolean constraints.

Personally, I'd like to let vgo play out a bit, I'm not convinced boolean
constraints are something I really want imposed on me by my dependencies.

More info for those who are looking for more context:
[https://research.swtch.com/vgo](https://research.swtch.com/vgo)
[https://github.com/golang/go/issues/24301](https://github.com/golang/go/issues/24301)

Whether you agree or disagree, its novel research in this field and excellent
technical writing worth reading in its entirety over a cup of coffee.

~~~
lobster_johnson
Thanks for the correction; I worded that unclearly. My point is that with MVS,
the solver will favour the oldest package that is allowed by the semver
version constraint; given transitive dependencies on multiple packages, it
will conservatively pick the oldest that satisfies all the constraint. You can
bump the version you want by updating the version constraint, but unlike other
tools (e.g. Cargo [1]), it will always favour the oldest allowed version.

[1] [https://research.swtch.com/cargo-
newest.html](https://research.swtch.com/cargo-newest.html)

~~~
IshKebab
Well, sort of. It's more like it has implicit versioning based on a lock file.

In other systems like Cargo and NPM you have a "lock" file which specifies the
exact current version of dependencies to use, e.g. if you say you depend on
"foo >=1.0" and when you build your project it actually downloads "foo 1.4",
this will be written to the lock file. When foo 1.5 is released, it won't be
automatically upgraded even though you said it should be fine.

vgo is exactly the same. When you first add foo1 it will implicitly (via
semver) assume you meant "foo >=1.0" and download the latest version - foo
1.5. That version will be written to the lock file (go.mod) and when foo 1.6
is released it won't be automatically used. Exactly the same as NPM and Cargo.

And just like NPM and Cargo you can trivially update to the latest compatible
versions of your dependencies with `vgo get -u` (like `cargo update`).

It's not that different. The main differences are you can't specify maximum
versions, and different major versions are treated as different packages so
you can easily have them both.

------
eknkc
For an introduction to vgo and to see the other side of argument, watch the
presentation from Russ Cox:
[https://www.youtube.com/watch?v=F8nrpe0XWRg](https://www.youtube.com/watch?v=F8nrpe0XWRg)

I was sceptical but now kind of sold on the idea after wathing the
presentation.

------
rosshemsley
Anyone have a bullet-pointed summary of the "crux" of what the author dislikes
about MVS?

I find MVS to be a very natural and simple solution to versioning - I feel
like it's much closer to how people actually manage their dependencies in
practice.

After an admittedly low-effort skim of this article, most of the arguments
seem to be that the author doesn't think it "feels right". Is there something
more concrete hidden in here?

~~~
cdoxsey
Its a long post, but I don't think he actually gives the reasons, rather he
intends to write several additional posts that will.

~~~
ngrilly
Honestly, Russ Cox published his first post about vgo on February 20
([https://research.swtch.com/vgo](https://research.swtch.com/vgo)). It's been
almost 3 months then. It's not fair to ask the community to wait any longer.
If Sam Boyer has good reasons to oppose vgo, then he should publish them now.

~~~
sagichmal
Russ waited over a year and a half before engaging meaningfully with the
community on package management, and waited another nine months before coming
up with this greenfield proposal. There's no reason to rush things now.

~~~
reificator
> _There 's no reason to rush things now._

Golang is hurting right now because of this issue. In fact this and the lack
of a canonical GUI approach are the only two caveats I have when suggesting
the use of Go.

I'm not saying to rush, but the constant churn in this space needs to end as
soon as it can. For a language with as much of a focus on simplicity and
stability as Go, it's embarrassing that this has been an issue for so long.

I have been using Go professionally[0] starting sometime between 1.1 and 1.4
and in that time I have something like 4-5 different ways of handling
dependencies in my repositories based on when those projects were started
and/or last overhauled. Each time I changed my approach I was following the
current best practices, or so I believed. It's madness, and it has to end
sometime.

[0]: Started playing with it in 2009, in fun projects I'll use whatever works
and/or is fun and/or gets the job done soonest. In professional projects I'm
much slower to adopt new tools.

~~~
sagichmal
> Golang is hurting right now because of this issue.

Go was hurting _three years ago_ because of this issue. The pain was tractable
and critical. The community made a thorough and good-faith attempt to end the
churn with `dep` -- which was summarily dismissed, ignored in part and whole.

I don't see why `vgo` should get the fast track now, when the core team has
been dragging their feet on the issue for years.

~~~
reificator
> _The community made a thorough and good-faith attempt to end the churn with
> `dep` -- which was summarily dismissed, ignored in part and whole._

I've used `dep` on a few projects and I don't mind it. If that's what we use
that's fine with me.

> _I don 't see why `vgo` should get the fast track now, when the core team
> has been dragging their feet on the issue for years._

I think you misunderstand my point. I'm not throwing my weight behind `vgo`,
I'm throwing my weight behind _making a long term decision of any kind_.

I get that it's important to take your time and get things right. That way you
can avoid starting down one path and wasting everyone's time, then switching
and declaring that everyone follow you down this one, and so on, and so on...

Oh wait - that happened anyway.

------
abiox
i think one fundamental problem is that for most ecosystems, versioning is
almost entirely arbitrary, manual and dissociated from language semantics.

the elm package manager seems to take a step in the right direction where
changing a function signature will force a version bump if you want to publish
the update. (iirc)

are there other languages/ecosystems that do something interesting with
versioning?

~~~
steveklabnik
We're working on similar tooling to that in Rust [https://github.com/rust-
lang-nursery/rust-semverver](https://github.com/rust-lang-nursery/rust-
semverver)

------
IshKebab
This badly needs a TL;DR. If there's something seriously wrong with vgo's
dependency versioning then it should be possible to explain it in less than
100k words.

~~~
TheDong
> If there's something seriously wrong with vgo's dependency versioning then
> it should be possible to explain it in less than 100k words.

That is a very lazy argument.

It's perfectly reasonable to claim something is flawed for numerous complex
reasons. It's equally reasonable to claim that a complex flaw may be difficult
to explain concisely.

Most importantly though, I don't think Sam is claiming that there's an obvious
fatal flaw in vgo, but rather that it is not the ideal choice given the
problem space.

To make that more subtle argument, it is necessary to fully define the
alternatives and problem space, which is what a lot of the words are
attempting to do.

The tl;dr would be along the lines of "vgo knowingly makes a set of tradeoffs
which I think are worse than the tradeoffs that are possible with another
hypothetical option, which I am also proposing"... but that tl;dr is not
really a very useful one, and attempting to condense a more meaningful one
will lose a bit too much nuance I think.

~~~
closeparen
The post could stand to have an introduction that lays out its arguments at a
high level, as is hammered into students at every level of English class.

