
Russ Cox on Go dependency management - aberoham
https://threadreaderapp.com/thread/1022588240501661696.html
======
kodablah
Long live reasonable BDFLs. They may annoy the community here and there, but
the cohesion and gatekeeping provided by a quality BDFL is hard to match with
a community. It's very hard for a community to have collective foresight, but
even when they do, often a gatekeeper emerges anyways during review. Of
course, it requires thicker skin on both sides of the coin, but in hindsight
the tech benefits often outweigh the politics.

Sure, some communities (e.g. the Rust one) show that a sole person need not
reside at the top. But that often comes with slow implementation/decision
costs.

~~~
pcwalton
Rust would have _died_ if it used the BDFL model. We had a huge disadvantage
starting out, with a tiny team at a smaller company and little name
recognition among the lead developers. The only reason Rust achieved any
popularity at all was that we listened to the community when they were pushing
us to compete 1:1 with C and C++ on performance. Cargo likewise would have
been terrible without community input. The community was starting hostile
forks before we course corrected.

Additionally, Rust development frankly moves faster than any other language
that I know of. You need only look at the release notes for that. The reason
is that people can truly be invested in the project, able to make technical
decisions as well as contribute code.

~~~
sbjs
Community input is different though. Communities need a leader (or small set
of leaders) who's responsible for judging whether something fits into the
overall vision of the product. If a whole community agrees that something
should be this way or that, then sure, that's darn good evidence that it's a
good fit and probably inherent to the usefulness of the product, and it would
be unwise for them to ignore it. But sometimes you have 50/50 splits and a
leader is there to make the ultimate decision based on their expertise and
understanding of the problem-domain and how the product fits into it.

~~~
pcwalton
That's why we have the small language teams who have to sign off on things.
It's not democracy.

I think the ideal situation is when the community develops a solution that
works well for them and then leadership steps in to shepherd it through the
integration process. Leadership isn't there to force their ideas onto the
community. When done properly, everyone is empowered and nobody comes out of
the process frustrated. This is something we have had to learn over time, but
I truly think it is the right way to develop open projects like languages.

------
nicpottier
I hope this apology and acknowledgment of the problems in the process appease
Sam. By and large he has been the one to driving the last resistance to
adoption of go modules and I can't help but think most of that is just
investor's dilemma.

At this point it would be great to no longer be talking about whether go
modules should be the thing but working together towards making them work for
everyone. (which I don't think is far off actually, Russ and co have done a
good job post-reveal of addressing use cases that break with modules)

------
ThePhysicist
I absolutely love writing programs in Go and at my current startup we're
basing all our core backend infrastructure on it.

One thing that I find extremely troubling about Go's dependency handling
though (and which at least partly contributes to people's frustration I think)
is the mixing of the concepts of a namespace and a resource identifier in Go's
package names. It's absolutely reasonable that package names should be unique,
making the package name double as a storage location is not always a good idea
though (IMHO): For example, we recently switched from using gitlab.com to our
own private Gitlab instance, which runs under a custom (non-publicly
resolvable) domain. Since all our packages were named like
"gitlab.com/.../..." (to make them "go get"-table) we would have to rename all
of them to make them "go get"-table from the new location. In the end we
solved it by defining a git URL rewriting rule that rewrites the old to the
new URL when fetching the repository. This ensured that we could work
primarily on our new Gitlab server while maintaining the possiblity to build
packages on the gitlab.com server. While that works I think it would have been
a better idea to keep package names and storage locations separate, or at
least give users a way to override/modify the resolution of packages into
locations using a more standardized approach: Such a mechanism could then
allow for different name resolution strategies (like a package file, a list of
standard repositories) while falling back to the standard way of resolving
packages into URLs for "go get". Granted, this would add another layer of
complexity, but as we already need a separate file for proper vendoring (to
e.g. specify versions) it wouldn't hurt too much to also put (optional)
information about storage locations into that file.

Just my 2 cents, apart from this minor inconvenience I'm absolutely happy with
Go and its ecosystem!

~~~
ithkuil
if you want to decouple the namespace from the repository location you can!

[https://golang.org/cmd/go/#hdr-
Remote_import_paths](https://golang.org/cmd/go/#hdr-Remote_import_paths) (read
until the end)

or TL;DR: [https://blog.bramp.net/post/2017/10/02/vanity-go-import-
path...](https://blog.bramp.net/post/2017/10/02/vanity-go-import-paths/)

~~~
TheDong
You still can't really.

If I have a canonical import path of "mydomain.io/foo", and I host html there
pointing to github or whatever, I now have a different single point of
failure: the domain. If I lose control of the domain or the

Most package management systems have one additional layer of resolution: a
centralized pluggable repository location which can be configured to some
default, and then lookups are made against that instead of against a url
specified in the source code.

For example, with cargo, I can open my ".cargo/config" file and specify
"registry.index = whatever-url.com", and then for package "foo" it'll look it
up on that registry.

If I forget to renew the domain, I don't have to change a bunch of import
comments in my source code to update the registry location; I have to update
one cargo config file.

~~~
Merovius
> If I have a canonical import path of "mydomain.io/foo", and I host html
> there pointing to github or whatever, I now have a different single point of
> failure: the domain. If I lose control of the domain or the

> Most package management systems have one additional layer of resolution: a
> centralized pluggable repository location which can be configured to some
> default, and then lookups are made against that instead of against a url
> specified in the source code.

TBH, these two seems to contradict each other. You don't want to use your own
domain, because that's a SPOF that you could lose control over. Whereas the
centralized repository is a SPOF that you never had any control over to begin
with. Seems strictly worse to me. And the nice thing about the chosen approach
is that it can coexist with what you seem to be wanting: gopkg.in exists and
is exactly that. A centralized, hosted (and free!) registry for go packages
with its own registration and authorization process.

What I find so beautiful about the go-get solution is a) that you can build
any other mechanism I know of on top of it - i.e. the user can decide on their
own tradeoffs of convenience/scalability/security/future-proofedness… and b)
that domain names already provide an elaborate, globally agreed upon namespace
that can separate ownership from administration and hosting, with its own
takedown-semantics… It's decentralized, existing, robust infrastructure. It
gives full control to the user. And it solves a whole bunch of technical
challenges already, for free. It means the Go team doesn't have to manage user
accounts of package maintainers and it doesn't have to have policies of what
code to accept or not accept - as it can't enforce them.

It's really quite brilliant, IMO.

~~~
TheDong
The point is that I don't have to change any of my source code in order to
change what registry I point at.

I can use "foo.registry.com" with no changes to my rust source code.

If I want to switch go from "old.com/x/package" to "foo.com/x/package" I have
to rewrite all the source code first.

That's the weird coupling which makes go not so configurable; code has no
place dictating where it's downloaded from, yet go packages do that. I can't
use the go package management ecosystem if my canonical import path doesn't
resolve (e.g. dep won't work, go get won't work, vgo won't work).

I never have to worry about my rust code having a magic comment in it which
makes the package manager incapable of fetching it from some arbitrary
differently named mirror. That's simply not possible in a well designed
package management system.

~~~
ThePhysicist
In our CI setup we ended up fixing it like this (which I still consider very
hacky though):

    
    
        git config --global url.https://our-new-domain.com/.insteadOf https://gitlab.com/
    

This allowed us to keep our code unchanged and still be able to build it on
different CI servers (e.g. our public as well as private instances).
Specifically, for Gitlab our replacement config looks like this:

    
    
        git config --global url.https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_HOSTNAME}/.insteadOf https://gitlab.com/
    

This assumes that all packages hosted on gitlab.com are available on our local
Gitlab instance, which can be problematic as well (for example if we would
import/use an open-source package hosted on gitlab.com along with our own
code). In that case we would need a replacement config that would only match
our groups/namespaces on gitlab.com.

I vastly prefer the Rust way of doing things though, as I don't think using
domains as a namespace is a good way. For example, most code hosting sites
will shut down or be deprecated eventually (possible even Github at some
point), which for Go will lead to a lot of headaches since there's no way to
know where a given package will end up when it's no longer available at its
original location.

------
hitekker
One meta pattern I’ve noticed: If team members can’t talk straight with one
another, the team can’t make good collective decisions.

Mealy mouthed walls-of-texts are just as unhelpful as one sentence dismissals.
Sooner or later, the silent miscommunication will manifest itself and impact
the product directly.

Needless to say, the health of a project relies upon the health of its
politics.

~~~
krageon
A lot of reasonably successful open-source projects (eg almost the whole npm
namespace) is rife with some breed of toxicity and nevertheless does "fine"
(for some value of fine - if you never look at it and ignore the security
breaches) in terms of the technical merit.

~~~
rauhl
I don’t think it’s just open-source projects: a great deal of _all_ projects
end up rife with some toxicity or another, and normally they do alright in the
end. Human beings are a problem: we have emotions, we have feelings, we have
needs, we’re not hyper-rational automata. But we mostly muddle through
alright.

I don’t know if we can avoid this. What I _think_ we _can_ avoid are blow-ups,
or at the least calm them back down again. We have to use our hearts, not just
our heads, to say and mean, ‘yes, I acknowledge your feelings, which are
valid.’ In the case of Go & dep, I think Sam Boyer feels let down because he’d
been so proud of the idea that his tool dep would become the official tool.
Rationally he acknowledged that it was the official experiment (maybe he even
acknowledged that was _an_ official experiment — I forget), but emotionally I
believe that he thought that it’d become the official tool. To see it replaced
is a colossal letdown.

We can honour him and his investment of time & energy while at the same time
acknowledging that the tool he created isn’t the right tool long-term. This is
something the net is terrible for: it’d be far better to break the news, in
person, over dinner or drinks. There’s no _good_ way to do it, of course, but
sharing physical space helps.

------
kevingadd
A common theme in this thread is the relatively typical software development
approach of 'well, I'll let them sort it out'. It's not lazy, and it sort of
resembles delegating problems you can't handle to someone else - but the
problem is if you do it enough on large scale projects you just end up with a
huge stack of miscommunications, and a talented team that has put together a
high quality product that completely fails to do what it needs to do. You
can't successfully delegate without ensuring that everyone knows what their
commitments are, and you need to pay attention on a regular basis to make sure
that everyone still knows what the plan is.

It's easy to end up in a position where you're making that mistake if you're a
small team used to shipping smaller products or features. Suddenly you're on a
bigger project, maybe a bunch of the developers work at another company, maybe
you've got a few million users instead of a smaller set like a hundred
thousand, and the same planning and team collaboration techniques you've used
are _subtly_ harmful and produce bad outcomes a year or two down the line. I
observed the WebAssembly design process suffering from same of the same
problems early on because it was a group of engineers from multiple companies
who were generally used to just working in tiny silos on experiments or
features of their own, not A Collaborative Effort For Millions Of Users, much
like what Go has turned into now that it's scaled out and not just a prototype
language made for Google internal use.

~~~
rurban
For me the important point was that the implementer cannot delegate the
community discussion to someone else. He needs to be part of it, and waste
hours of important development time on bike shedding.

Or just ignore it and apologize afterwards for rendering the bike shedding
useless.

~~~
sgift
Yeah, it is bike shedding to discuss dependency management, because dependency
management is an "insignificant or unimportant detail" of a programming
language. /s

~~~
majewsky
You say "/s", but for some reason, many people still hold the sincere position
that dependency management is something that can just be tacked on to a
language as an afterthought.

~~~
coldtea
Of course it can. npm, composer, etc all added it to their respective
languages as afterthoughts, and many years (or decades even) after the
language was out.

~~~
al2o3cr
Given the whole mjs-vs-js situation with npm, it's more of a counterexample
for adding dependency management late.

~~~
coldtea
That's nothing really. They have to use 2 different suffixes to denote 2 kinds
of files vs modules.

Meanwhile millions of people use it for 10 or so years for all kinds of
deployments.

If only all languages had that unimportant problems.

------
keypusher
> Dep does not support using multiple major versions of a program in a single
> build. This alone is a complete showstopper. Go is meant for large-scale
> work, and in a large-scale program different parts will inevitably need
> different versions of some isolated dependency.

This seems to be a major sticking point but I'm having trouble understanding
why. I'm not familiar with any other package management system which supports
simultaneous major versions in a single project. If two things require
different versions of a package, those seem like separate projects to me. Is
this to support dependencies which rely on the same package at different
versions? If so, why not just call it a version conflict and fail until it's
resolved?

~~~
derefr
> Is this to support dependencies which rely on the same package at different
> versions?

Yes.

> If two things require different versions of a package, those seem like
> separate projects to me

Basically we’re talking about libs that have version conflicts, but also very
slow release schedules, such that the version conflict isn’t going to be
reconciled any time soon; or where one of the deps may even be effectively
abandonware, but “works just fine” stand-alone so nobody’s going to update it.
It just depends on extremely old versions of common shared deps. These are,
effectively, _irreconcilable_ version conflicts.

The only thing to do, in most languages, is to fork one or the other dep to
fix the conflict; or to split the project into microservices such that the
conflicting components can live on opposite sides of a process memory
boundary. Even these are sometimes impossible for office-political reasons.

> I'm not familiar with any other package management system which supports
> simultaneous major versions in a single project.

Node’s npm. Versions aren’t globally resolved; instead, each package
effectively gets its own version resolution. Your root-level package gets a
node_modules/ directory with your direct dependencies checked out into it; but
your deps' deps are checked out into further node_modules/ directories that
exist within their parent dep's checkout directory. It's somewhat as if each
parent dep had vendored its deps, and had a vendor/ subdirectory; except that,
in this case, the vendoring process is recursive.

When a given package require()s a library by name, then, it’s actually
importing its own npm-vendorified copy of the library from its own
node_modules/ directory, rather than looking in your package's root
node_modules/ directory. (In fact, there is no root level; you have "package
isotropy", where a dep NPM checked out has the exact same NPM-populated
subdirs as a project you git clone'd yourself.) Since these libs aren’t
entered into a global namespace by require(), rather just have the module
returned as a runtime object for the parent scope to bind to a local variable,
there are no runtime conflicts between the separately-vendored versions,
either.

In a language without such runtime support, though, you would need your
package-ecosystem tooling to reach into your downloaded lib and do some name-
mangling of the exported symbols, and then either create a router symbol that
resolves calls based on the lexical scope, or further mangle your deps'
parents to call the dep by its mangled name rather than its original name. I
_think_ that’s what’s being proposed here, since Go has no such runtime
support. (Effectively, in Go terms, you can simply think of it as all the
references to git-http URL depspecs in import statements, being amended to
qualify the repo name with the full git SHA, such that two different commits
of the same project act as if they were two differently-named projects from
Go's perspective.)

~~~
rpeden
It's a relatively minor nitpick, but since version 3, npm flattens the package
tree as much as it can and tries to bring everything possible to the top level
node_modules directory. It'll still use nested node_modules where necessary to
handle dependency version conflicts. A bit more info here:

[http://npm.github.io/how-npm-works-docs/npm3/how-
npm3-works....](http://npm.github.io/how-npm-works-docs/npm3/how-
npm3-works.html)

------
StevePerkins
If you worked with Go quite a bit 3-4 years ago, but have been out of scene
since then... what is the best resource(s) for getting back up to speed with
its current state?

I presume that the language spec itself hasn't change DRAMATICALLY over the
past few years. But so much is unfamiliar in the project structure / build
process area (e.g. "vendor directories", modules, dependency managament, etc).

All the tours and tutorials focus on the language syntax and standard library.
Looking for a refresher or intro to the "how do you actually get work done in
the best manner?" plumbing aspects.

~~~
swah
I'd say: assume nothing has changed. Take a look at the active projects in
your area of interest. A few ones:

[https://echo.labstack.com](https://echo.labstack.com)

[https://github.com/go-kit/kit](https://github.com/go-kit/kit)

Web glue frequently recommended around here:

[https://github.com/go-chi/chi](https://github.com/go-chi/chi)

Github pages like [https://github.com/avelino/awesome-
go](https://github.com/avelino/awesome-go) have become a good starting point
for finding interesting projects.

------
privateSFacct
One issue I have is I can't really find the dep complaints about go's approach
clearly. There was supposed to be a talk but I can't find the video.

Yes - Russ should have been much clearer much earlier and not let dep get as
far along as it did. But dep mentions the issues, particularly for small teams
with go's new approach - I'd love to get a quick version of that.

~~~
Merovius
The closest to that is
[https://sdboyer.io/vgo/intro/](https://sdboyer.io/vgo/intro/) and
[https://sdboyer.io/vgo/failure-modes/](https://sdboyer.io/vgo/failure-
modes/). IMO these posts are hard to digest and ultimately unconvincing. But
YMMV.

------
shadowmint
I think I’m more offended Russ said he was looking at cargo as ‘best of breed’
for inspiration.. but ended up ignoring it entirely to do his own thing.

Just suck it up.

You went off and did your own thing, and ignored everybody else, sticking
pretty much to what you wanted to win the race in the first place (gopkg).

Right, now thats out the way can we just get on with it and use modules?

Decisions have been made, they’re here to stay. It doesn’t make everyone
happy... but hey, _we’re all_ massively relieved the mess has been sorted out,
so thank you.

Lets stop these deep soul searching what he said she said wastes of time and
get get started using it~

~~~
skybrian
It's possible to study another product and learn things from it while still
deciding to do it your own way. Deciding to do things differently is not an
insult. It's one way innovation happens.

------
jatins
This is kinda unrelated. But I am _very_ new to Go.

If one were to start a serious project with Go in 2018, what's the recommended
way? Are people using Go modules? How to deal with 3rd party deps which dont
have go.mod files?

When I previously explored Go, having all code under GOPATH was recommended,
but seems like things have changed with `vgo`.

~~~
ben0x539
Modules are still in a beta pre-release kinda state, I think most people would
stick to dep even for new projects right now.

~~~
kaeshiwaza
Dep has also issues, it's safer to continue to use Dep when you already use it
but for new project it's recommended to switch now to go modules. Anyway, most
of the time it doesn't change anythings to lock the version with dep or use
minimal version selection.

------
ckastner
Off-topic rant: I really hate how Twitter has popularized this devolved style
of communicating by a flurry of fragments _so much_ that we now have services
like Thread reader (which I appreciate!) to undo that mess, and to make
communication coherent again.

~~~
nicpottier
Ya.. this is an especially funny case because I can't imagine Russ wrote this
adhoc on Twitter, this looks like something very much prepared, then dribbled
out, which makes me a bit sad for the world.

~~~
gant
Twitter has that feature where you can post multiple tweets in a chain.
Honestly, why have a character limit at all at this point...

~~~
sigjuice
So it is accessible over SMS?

~~~
greiskul
How many users actually use it over SMS though?

------
ainar-g
It's _Russ_ Cox, not Ross. Can the admins change the title?

~~~
innagadadavida
Calling him out like this is pretty bad, why not make this about the Go
dependency management instead of one individual?

~~~
coldtea
Because it's a long twitter of messages that he posted discussing Go's
dependency. It's not a general discussion between various people, it's sorta
like he wrote a blog post on the matter.

