
A response about dep and vgo - Zariel
https://peter.bourgon.org/blog/2018/07/27/a-response-about-dep-and-vgo.html
======
Athas
Vgo has certainly been a disaster from a communication and community point of
view. From an outsider perspective, it looks like a case of maintainer
arrogance, and it _is_ hard to believe that Go really has needs that are
significantly different from those of e.g. Rust or Java.

However, I am also in the unusual position that I had to design and implement
a package manager for my own little language recently. My original hunch was
to just create a Cargo clone, but after reading Russ Cox's writings, I ended
up cloning vgo (with a few simplifications). The implementation was
wonderfully simple - everything just fit together. The algorithms are trivial.
Operationally, it is easy to understand what the package manager is doing
(even more so than for vgo, as vgo has do deal with various Go idiosyncracies
and backwards compatibility).

This is in stark contrast to my experience with other package managers, which
are temperamental beasts at the best of times. When they work, everything's
groovy, but their error modes are easily incomprehensible. I think vgo's
approach of restricting expressivity and streamlining processes is the right
one. But since nobody has used such a package manager before, it remains to be
seen whether it works in practice.

For my own experiment, I do have one data point: I showed people the (rather
brief) documentation for my vgo-inspired package manager, and they all felt it
was very simple to follow and easy understand what it did, and how.

~~~
zeeboo
For what it's worth, I analyzed as many Gopkg.{lock,toml} files I could find
in the wild, and found that vgo's algorithms would service all of them. I've
done similar analysis for a smaller number of Rust projects using Cargo and
had the same findings. I've found it hard to find projects in the wild that
actually do any non-trivial version selection. I think this is some good
evidence that it will work out in practice.

I think it's less that Go has significantly different needs, but it's more
that people overestimate what their actual needs are.

[https://github.com/zeebo/dep-analysis](https://github.com/zeebo/dep-analysis)

~~~
pcwalton
The problem with minimal version selection isn't that it can't support
nontrivial version selection but rather that it doesn't automatically select
the newest versions of packages.

~~~
kaeshiwaza
It's a feature not a problem. It select the only know working solution. You're
still free to update to newest versions.

------
nemo1618
I'm relieved to see that the fate of Go is ultimately controlled by a small
team of highly talented engineers, not a nebulous "community" that designs by
democracy.

If Go had been designed by community vote from the beginning, it would almost
certainly have generics... and operator overloading, and exceptions, and 50
exposed GC knobs, and macros -- and a SAT-based dependency resolver, of
course.

I trust Russ, Rob, and the rest to get it right, and they've proven that my
trust is justified over and over again.

~~~
ngngngng
> nebulous "community" that designs by democracy

I can't read that statement without thinking of perl, and the nightmare of
developing in a team of perl devs. We don't need ten thousand ways of doing
things and not everyone needs their way to be represented. I'm very happy to
have Russ enforce one "right" way of doing things and vgo seems to fit that
nicely. One less argument for me and my fellow engineers using go to argue
about.

------
jsd1982
I am not involved in this discussion and have no skin in the game, but from an
outsider's perspective it seems that Russ laid out exactly what his concerns
were about showstoppers in the dep implementation and those concerns were
effectively ignored by the dep committee. What were they expecting to happen
after that? Are there some other non-showstopper classified concerns that Russ
should make up on the spot and have the dep committee then concentrate on? Why
bother after showstoppers are effectively ignored?

It's odd to call out Russ as being disingenuous when he says his concerns were
ignored when the dep committee did effectively ignore them. Adding
'effectively' as a qualifier here is somewhat important because it grants that
there may have been a deliberation process and that those concerns were not
outright dismissed, but the output of a deliberation process must be either in
favor or against, and when a concern is deliberated against it is effectively
ignored. It has never made sense to me when someone says something along the
lines of "we've heard your concerns and have taken them into consideration"
and yet did not heed those concerns. Obviously you haven't taken them into
consideration and therefore they were effectively ignored.

In this case it seems to me that the burden of proof is on the dep committee
to demonstrate that the alleged showstoppers are in fact not stopping the
show. You don't get to call a committee meeting, decide to ignore showstoppers
by claiming that you don't believe them to be so, and then expect nothing but
smooth sailing and cooperation from there on. That makes no sense at all. All
this rhetoric about community involvement and "working with us" falls on its
face when the implicit terms of engagement are not adhered to.

~~~
h8liu
I think the issue is that Russ should have clearly told the dep people that if
the showstopper issues are not handled, then he will seek for some other
solution, e.g. build one by himself.

However, what happened was that he told the dep people "I will build a dep
tool myself to understand more on the problem details", which implies "after I
understand more, I will come back to you and discuss more", but not "after I
build the tool myself, I will just integrate my version into the official
tool". For this part, Russ was not executing his own words, or at least
delivered the wrong impressions. Maybe this result should be already well
expected based on Russ's personality or how the golang team did stuff in the
past, but many programmers are not that politically awareing..

It is not the technical result that the "vgo" solution kills "dep" saddens dep
people. It is rather the form of communication: "vgo" stabs "dep" in the back,
by emerging suddenly out of Russ's pocket, and essentially claims immediate
victory using the power of the Go core team. When the vgo posts are out, and
the vgo proposal is posted (and it is not really bad technically), there is
really no effective way for the rest of the community to reject it.

Russ could have kept his words by going back to the dep people and told them
about what he learned from writing "vgo", and then move forward with
integrating "vgo" if a consensus cannot be reached. Eventually, Russ might
just be able to get the same technical results as today anyways (but maybe
with larger communication overhead).

What the dep people essentially says here is that, if Russ really wants, they
can also implement exactly "vgo" from "dep", but they never got the chance.

~~~
jsd1982
If I were in Russ's shoes and I had told someone as politely as I could that
their solution exhibits what I consider showstopper issues, I wouldn't expect
to have to clarify that I find that solution entirely unacceptable and flawed
from its very design. There's no issue in the clarity of communication when
the term 'showstopper' is thrown around.

Russ said he was going to go and build a tool to understand the problem
better. In no way does that imply that any lessons he learns from the
implementation must be communicated back to the dep committee. But for the
sake of argument, let's say that did happen. Then what? Does Russ practically
force the dep committee to implement vgo out of the remains of dep? Why would
he do that when he just spent a month or two implementing vgo from scratch?
Why bother going back to a problematic group that doesn't believe their core
issues are showstoppers? If that difference of perspective exists and they're
not willing to debate that, to say nothing of the fact that they feel that
they should be on the receiving end of the burden of proof, what more can be
gained from them from Russ's perspective?

~~~
h8liu
hmm.. I was not aware that Russ used the word "showstopper" until recently in
his tweets, and even if he literally used the word in the past, the word of
the meaning was not delivered, reading from the post here.

That said, the dep people also made a similar mistake on not clearly
confirming with Go core team that they had a chance to integrate dep into the
official toolchain eventually. They wanted dep to be the thing, and they
really thought that they were on the right track.. poor folks..

If Russ managed to influence the "problematic" group to do what he wants,
rather than doing it by himself, he could give these people a meaningful place
in the Go community. _That is the gain._ If that is worthy or not (comparing
with discarding 2-month work on Russ's own time) depends on your own
perspectives, and I am not sure what Russ thought or thinks. No matter how, it
will affect how the community grows in the future.

So this goes back to the root issue about how the Go core team view its
Google-external language community (or even more fundamentally, if they will
be rewarded/recognized by Google if they did a good job on making the
community happy). Specifically, do they want to carry the responsibility to
deliver messages to the community clearly, at least to some certain extent?
Opinionated is fine and firing users (e.g. users who think generics is a must)
is fine, sometimes even preferred, but hurting people that were willing to
follow and contribute might not be the best way to go.

------
ereyes01
Rather than a technical argument, or a clarifying point of view, all I see
here is basically "we formed a committee goddammit, we're the incumbent, how
dare you not listen to us?"

In my eyes, Russ is too nice here, and this committee seems like it would've
been difficult to work with at an intellectual level, anyhow, judging by this
type of griping.

Maybe Russ should hire Linus to deal with situations like this :-)) /s

------
coldtea
> _Community consensus is not always possible. If we don’t get there, then the
> core Go team decides. Technically I am the final decider but what actually
> happens is that a bunch of long-time Go team members talk through the
> decision to get to a consensus among ourselves._

Was anything major ever decided based on a "community consensus" in Go?

What I've read about, and this case is no exception, is that it's always "a
bunch of long-time Go team members". Aside from making libs, and conferences,
and such, the community might as well not exist.

Certainly nothing like PEP process, or something like Swift (which despite
Apple's thing, goes out of its way to engage the community in feature roadmap)
or Rust.

> _We did all of this because we wanted to be an exemplar of how the community
> could step up and solve a problem that was being ignored by the core team. I
> can’t think of anything else we could have done to be better than we were.
> But the end result of this effort was no different than if we had done none
> of it at all: the core team ultimately didn’t engage with us meaningfully on
> the body of work we’d contributed, and instead insisted on doing the work
> themselves, as an essentially greenfield project._

Well, have they had any other impression about the prospects of such external
effort? The author adds an even more bitter remark later on, and then retracts
it saying: "Upon reflection, I think this may be too strongly stated. There
are good examples of large contributions that originated outside of the core
team, including support for different architectures, like WASM".

But adding a different architecture is not a decision that changes the
languages syntax, standard libs, tooling, direction or semantics. It can even
be dropped at any time, and none other architecture would even care. It's like
proving how an OS is "open" to third parties by pointing out that anybody is
welcome to contribute a hardware driver for it.

~~~
geodel
> something like Swift (which despite Apple's thing, goes out of its way to
> engage the community in feature roadmap)

Good you say that. Not too long ago solutions from swift-server-group is
discarded and Apple simply developed swift-nio in-house. I read that server
group was blindsided by this development and now more or less disbanded.

~~~
coldtea
Not sure about the swift-server-group situation -- some where blindsided, but
here's e.g. the head of the closely related Vapor (server side web framework)
project:

"I've seen a surprising amount of people thinking that the server-side Swift
community or the Swift Server working group was somehow blind-sided by
SwiftNIO. That couldn't be farther from the truth. We had known about SwiftNIO
before the first line of Vapor 3 code was even written."

And in any case, even if the server group was blindsided it's not
representative of a general tendency, as there are a very active features
discussions e.g. in
[https://forums.swift.org/c/evolution](https://forums.swift.org/c/evolution)
with the community and public members involved and shaping changes.

~~~
geodel
Well discussion is all fine. The important thing will be if Swift core team
which is all Apple employees (except Chris Lattner who left Apple recently and
removing him would look petty) will simply accept community solution even if
they don't think its right.

I don't know of examples if anything got in Swift just by force of community
without core team approval. And it seems right thing to me whether Swift or Go
or Rust.

One thing better in Swift is clear rejections of pitches if it is not right
for swift which makes sense as communication is always better with everything
Apple.

~~~
coldtea
> _I don 't know of examples if anything got in Swift just by force of
> community without core team approval._

It's not about that though. Of course the core team should approve. It's when
the core team only approves its own things, which often are designed decidedly
not the way that the community wants, that's the problem.

------
jessaustin
_" Dep does not support using multiple major versions of a program in a single
build. This alone is a complete showstopper. Go is meant for large-scale work,
and in a large-scale program different parts will inevitably need different
versions of some isolated dependency."

Russ has asserted this from day one, and has brought several examples out in
evidence, but has simply not convinced the committee that it’s true. Calling
it a serious problem, let alone a showstopper, is a significant overstatement.
The strongest claim that can be made on this point is that it’s a matter of
opinion._

OK that's enough for me. "BDFL" clearly correct, random critics totally
unwilling to consider the proposition. More language developers are realizing
that hierarchical imports with directory name conventions are a good practical
thing that just works. Multiple versions come for free with this arrangement.
If your tool can't handle that, your tool sucks.

------
Confiks
The discussion about the original post can be found here:
[https://news.ycombinator.com/item?id=17623023](https://news.ycombinator.com/item?id=17623023)

This response is a long one, of which the ending deserves special note:

> I hope this story serves as a warning to others: if you’re interested in
> making substantial contributions to the Go project, no amount of independent
> due diligence can compensate for a design that doesn’t originate from the
> core team.

> Maybe some of you are reading this and thinking to yourself, “well, no duh,
> Peter, that should be obvious.” Maybe so. My mistake.

It personally made me think of this issue / PR in the Go issue tracker,
[https://github.com/golang/go/issues/20654](https://github.com/golang/go/issues/20654),
about support for constant time arithmetic with bigints. Currently, operations
on bigints in Go may leak secret data via timing channels, because
calculations with different values take predictably different time. The
maintainers have chosen to specially modify only the P256 curve (but not P224,
P384, or P521) to work in constant-time.

The author of the issue has written an extensive (somewhat strawman) PR that
includes the constant-time code _inside_ the current implementation of the
bigint. That's a choice – for seemingly good reasons laid out in the issue,
and of which the author admits that it also could be a separate package – but
the rest of the conversation is halted in indecision about how the core team
should proceed, without referring back to any of the arguments and reasons the
original author has put forth.

~~~
tptacek
I don't see the comparison you're drawing here. The question proposed in the
Go issue is whether the standard bignum library in Go should be constant-time,
whether crypto primitives should provide their own constant-time math, or
whether a third library for constant-time bignum math should be added that all
Go crypto can rely on. That's not an easy question, and meanwhile significant
chunks of the bignum crypto code that people actually use in Go is written in
assembly anyways.

The Go crypto libraries have a domain-specialist owner: Filippo Valsorda. If
you have questions about constant-time math operations in Go, he's a smart and
nice guy; you should maybe ping him.

Go's crypto library is imperfect, but it is the best native crypto library
provided by any modern language. (Ring is excellent too, but is not as I
understand it part of the Rust standard library).

~~~
c2h5oh
> I don't see the comparison you're drawing here.

I think the similarity is as follows:

\- there is a desire to fix something that feels like a gap (in this case
limited support of constant-time math in standard library)

\- there is a change proposal coming from the community

\- there are arguments why it should be included and some discussion if it
belongs in a separate library or not

\- very little or no input from core team

\- the whole process is stalled waiting for core team to participate

Now we just need someone on the core team decide they need to research via
implementation instead of actively engaging community, end up liking their own
solution more and we'll have dep all over again.

~~~
mbreese
> which the author admits that it also could be a separate package

I don't think the similarity is all that apt.

If the core team is the limiting factor in the PR, and if there is a need for
this, then why not just make the separate package and be done with it? If it
is popular and core wants to merge it in at a different time, then great.

~~~
c2h5oh
I'm not aware of a way to plug your own crypto to a lot of things that are
part of standard library and I'm not aware of a way to plug your own math
library to existing crypto.

~~~
jessaustin
Would they take PRs for adding that bit of plugability?

~~~
tptacek
Probably not.

~~~
helper
There has been some discussion about a pluggable TLS stack:
[https://github.com/golang/go/issues/21753](https://github.com/golang/go/issues/21753)

------
DannyBee
There's an interesting discussion about this with russ and peter participating
over on reddit here:
[https://www.reddit.com/r/golang/comments/92f3q1/peter_bourgo...](https://www.reddit.com/r/golang/comments/92f3q1/peter_bourgon_a_response_about_dep_and_vgo/)

------
uluyol
One thing that I think is being missed is that code/a product is not the only
thing that matters. In fact, it's one of the less important things here.

Once a design is set, implementing it is "just work". I'm not sure trying to
undervalue that, but the important part is first investigating the design. The
package management committee came up with the tried and tested design while
Russ was trying to understand the true requirements. Both have their merits,
and time will tell if Russ is right here. The thing to do keep in mind is that
investigative work is inherently "wasteful".

------
samat
I don’t have extensive knowledge of the matter in hand, but this narrative
seems much more coherent. And sad for Go/Google. So much for “open”. Just like
Android/Google play services thing.

Calling something “open” a hundred times a day does not remove one bit of
corporate control.

~~~
geodel
If someone is putting >90% of funding for project, it is just common sense
they will veto things they do not like. I wonder if some people believe that
online cliche about open source is all there to it.

------
sandreas
Is there any package manager using P2P technology (e.g. Bittorrent) to speed
up transfer? Or would this generally a bad idea?

------
ajcodez
Dep is awesome from the user perspective. You just type > dep ensure and it
works and it’s fast.

~~~
mappu
So is GO111MODULE. You just type `go build` and it resolves dependencies as
part of the build. It's even one less step than `dep ensure`.

------
0xdeadbeefbabe
Following the meta discussion is more taxing for me than following the
technical details.

------
claydavisss
Go does not exist to raise the profiles of Sam Boyer and Peter Bourgon. Sam
wanted to be a Big Man On Campus in the Go community and had to learn the hard
way what the D in BDFL means. The state of dep is the same as it was before -
an optional tool you might use or might not.

Lots of mentions in Peter's post about things the "dep committee" may or may
not have agreed with. Isn't this the same appeal to authority he is throwing
at Russ? When did the "dep committee" become the gatekeepers of Go dependency
discussions and solutions? Looks like a self-elected shadow government, except
it didn't have a "commit bit". Someone should have burst their balloon
earlier, that is the only fault here. Russ, you are at fault for leading these
people on.

Go is better off with Russ's module work and I personally don't care if Sam
and Peter are disgruntled.

~~~
zellyn
I've interacted quite a bit with both Sam and Peter. Your characterization
rings completely false. I've never gotten the least hint of the impression
they were doing this for self-promotion, but rather to serve to fill an
obvious vacuum, perhaps even with a degree of reluctance.

I remember being excited when the “dep committee” was formed, because (if I
remember correctly) it was set up with the explicit blessing of the Go Team,
with a Go Team Member on it to facilitate two-way communication.

Your characterizations are both inaccurate and ugly.

~~~
geodel
> Your characterizations are both inaccurate and ugly.

I think it may be blunt but seems true to me. Sam calling integration of Go
module as sad milestone does seem disgruntled.

Dep folks have decided not to take high road which is fine but others are free
to call this out.

~~~
weberc2
I want vgo to succeed and I like most of your contributions, but comments like
this and the original post (“Sam just wants to be the big man on campus”) are
petty and rude. You’re better than this.

~~~
geodel
Not that I disagree with you. But this unending whining by dep folks is not
helping them endeared to me or so many others.

