

Problems with Go Get - chiwawa
http://0x74696d.com/posts/go-get-considered-harmful/

======
aikah
It's clear to me that Go didn't get package management right,go get doesn't do
package management at all. I think today it is the most important thing the go
team has to solve.Otherwise, people will end up using different incompatible
solutions.

Projects need a manifest so that proper dependencies are declared. People who
say "just use make",or "just use git submodule" or "X feature of your VCS" are
just pushing for more fragmentation. I download a package.Now I have to check
what VCS feature it uses, wether it uses a shell script, make , or build tool
Z to fetch dependencies, do people advocating for these solutions really want
to use a language where libraries aren't compatible with each others? and then
some people say : just vendor and commit dependencies. Ok but let's say I find
a bug in one dep. Patching all the different versions of the same dep is just
not something manageable.

I think it's time go devs acknowledge the fact that there is an serious issue
here, instead of resorting to the ususal "you don't need that in go".

Godeps is a first step, but we need a official way to manage packages.So
everybody is on the same line.

I would really like something like composer for Go, with flat dependencies.
Which means that, unlikes NPM that downloads the entire world each time one
installs a library, dependencies are flat,which keeps API stables and reduce
the the amount of useless libraries and forks.

~~~
nothrabannosir
_> I think it's time go devs acknowledge the fact that there is an serious
issue here, instead of resorting to the ususal "you don't need that in go"._

I'm out of the loop, so honest question: does golang-dev still say that? I
thought they were not in denial of the problem, but indecisive about the
solution. Like with generics? Or do they consider this a non-issue?

I know Cheney recently came out with his own solution, so it can't be that
bad, right?

~~~
xienze
> I know Cheney recently came out with his own solution, so it can't be that
> bad, right?

Yes, and so have others. What they really need to do is say "this is our
blessed solution" and bundle it. Otherwise, there's just a bunch of competing
solutions floating around and nothing is really going to take hold.

And yes I'm well aware that Java doesn't have a dependency manager but Maven
emerged as the de facto way to handle dependencies. I think that has a lot to
do with developer attitudes. I see Go developers as much more "don't tell me
what to do, I'll roll my own solution" than Java developers, and that's why I
fear a third-party dependency manager won't win out in the Go world.

------
munificent
I'm awkwardly patting myself on the back here since I'm one of the co-authors,
but I really wish more people knew about the package manager we have for
Dart[1]. Really, though, I can't take credit, since we basically just do the
same thing Bundler does.

[1]: [https://www.dartlang.org/tools/pub/get-
started.html](https://www.dartlang.org/tools/pub/get-started.html)

It solves every single one of the problems listed here. It has a very simple
workflow:

1\. You make a pubspec.yaml file to list your package's immediate dependencies
and the version ranges you allow for them.

2\. You run "pub get". It finds all of your transitive dependencies, picks
versions that satisfy the constraints, and downloads them.

It also creates a pubspec.lock file that specifies which versions it picked
for everything. You check that into source control and now everyone on your
team will use the exact same versions of all of your package's dependencies.

3\. When you want to get a new version of a dependency, you run "pub upgrade
<package>".

4\. To add or remove a dependency, just edit your pubspec.yaml file then run
"pub get" again.

We have a central repository where packages or hosted, and you can also pull
in packages from Git or your local file system.

Packages do not at all interfere with each other. Two packages on your machine
can use different versions of the same dependency without any problem.

Bundler is an absolutely brilliant package manager. My prediction is that
every language either has a Bundler-style package manager or will eventually
move to one.

~~~
hittaruki
One of the thing Cheney says in his talk about gb is that nobody likes the
.lock files[1], I see that dart is using .lock files, bundlers also uses .lock
files as does rust.

But other languages seems to get away without it.. clojure with lenigen seems
to do it nicely.

So what are the problems with it?

[http://go-talks.appspot.com/github.com/davecheney/presentati...](http://go-
talks.appspot.com/github.com/davecheney/presentations/reproducible-
builds.slide#16)

~~~
Jweb_Guru
A slight tangent, but as someone who is sort of in the "respect but do not
use" Clojure category, Leiningen seems pretty amazing. I really hope future
language package managers look to it for inspiration; it seems to get
practically everything right.

------
jameskilton
Go's Dave Cheney is taking a different approach of solving this problem with
"gb". He agrees that "go get" should not be used.

[https://github.com/constabulary/gb](https://github.com/constabulary/gb)

~~~
andrewchambers
I think gb might work, but it needs to work in conjunction with a solution
like ruby bundler or dart pub imo.

~~~
gecko
It does, unless I'm really missing something. Since I'm not familiar with pup,
can you walk me through what of bundler's functionality you'd be missing?
Maybe I just don't use bundler correctly.

~~~
andrewchambers
gb doesn't know which ranges of versions for dependencies and transitive
dependencies are compatible with each other.

------
skj
Lately I've been working on something to address the problems that I see with
go get.

[https://github.com/skelterjohn/wgo](https://github.com/skelterjohn/wgo)

It's a workspace layer on top of the go tool, with added support for
repository pinning. You 'wgo get' the repo you want, and 'wgo {save,restore}'
will pin/fetch it in the future. Also, 'wgo FOO' will do 'go FOO' with GOPATH
set for your workspace, so all the normal goodness is still there.

It also makes it so you don't have to manage the GOPATH env var anymore, which
I always found to be a pain.

------
kjksf
While I do agree that go doesn't come with built-in solution for version
pinning, this hating on go get is misguided.

Most package managements solutions that are popular in other ecosystem (pip
for python, npm for node, cpan for perl etc.) combine 2 functions: downloading
code and versioning.

go get only does the first part: downloading the code. If you ask me they
solved the problem better than pip/npm/cpan by getting rid of intermediary and
going straight to the source.

go get has never claimed to be versioning solution so while I understand why
people make shallow comparisons to pip/npm/cpan and assume that it does and
complain that it does it badly (including this article), the reality is that
go get is not a versioning tool at all.

There are good reasons why versioning is not supported by default in go: there
isn't one way of doing it that is clearly superior to other ways. That's why
we have so many solution (including the one advocated by this article) and
none of the solutions became a de-facto community standard.

In summary: stop complaining that go get doesn't do versioning. There are
plenty of working versioning solutions to choose from, pick the one you like
the best.

~~~
DAddYE
This is the problem with most of the Go community: Blind.

There are plenty of tools that works quite well: bundler, cargo (even with
some young rough edges) and at some degrees npm.

So there are tools and better ways to do it.

It is just that Sir. Pike doesn't like them and Google doesn't need them.

> In summary: stop complaining that go get doesn't do versioning. There are
> plenty of working versioning solutions to choose from, pick the one you like
> the best.

The reason why there are plenty of them is just one: none of them works well.

Exact opposite of what you said. You don't have such mess of pkg managers in
others languages.

~~~
Dobbs
`go get` and `godep` have been excellent for us. It does everything we need it
to and have had no issues.

It has had far less issues than bundler, npm, etc.

~~~
DAddYE
I really doubt that. I always had problem with godeps at the point that I hate
it.

Right now for example I have a giant repo with tons of dependencies and:

    
    
       godep: unable to detect version control system for code.google.com/ path
    

Which dependency? Tried some and they work. -__-

------
thruflo
It's a meta point but the semantic of `go get` is so beautiful I'm profoundly
disappointed that the tool isn't usable.

Being able to dispatch a co-routine with `go ...` is beautiful. Being able to
fetch a dependency with `go get` is just as succinct and expressive.

I'm not up on Go language development but imho the world would be a better
place if they could fix the behaviour to preserve the syntax!

~~~
TheDong
Why is the semantic of `go get` any better than `bundle install`?

There's obviously no way they can fix the behavior in a backwards compatible
way because it is so fundamentally broken, and I think that doing something
because of a pretty name is absolutely ridiculous when it comes to creating a
technical product. You need names like 'heartbleed' if you want to talk to
management, but go get is a fundamentally developer focused tool and thus its
utility should come over its naming.

I almost see its naming as a bad rap against google developers because it's an
un-googleable phrase and who cares if it sounds pretty if it's un-googleable
and also utterly terrible.

------
bichiliad
True story: I was in a group project in a distributed systems class, and we
had made a project that relied on a go http routing package (I think it was
martini or something). For some reason, my partner's computer wasn't able to
present our project, so at the last minute, we cloned it to my computer.
Between then and the night before, the router changed it's API, and forced us
to do some crazy last-minute adjustments. In retrospect, we probably could
have manually reverted that one package's repo, but git is scary to do things
with at the last second.

TL;DR: Versioning is important.

~~~
papaf
As I understood, the original idea with go gettable libraries was that:

    
    
      - All releases go on master. Dev goes on separate branches.
      - Do not make backwards incompatible changes. 
        If you need to make big changes, make another project.
    

A lot of libraries ignored the advice. Whenever a library breaks its API (sqlx
for example), I stop using the library. This has made go get usable for me.

------
tracker1
What was really difficult for me to wrap my head around was how go
dependencies actually work out... via gopath, etc. My first exposure was the
bash scripts for a few projects, that I was trying to build in windows... and
it was to say an interesting first exposure.

I kind of wish that if go is going to use github targets for dependencies,
that they at least support semver pinning and require tagging in github to
support this. There are other issues, but I think that would help a lot.

It's always going to be an issue...

\----

Of all the developer package managers I've used npm is probably the best...
and even that has some issues.

    
    
        1. Platform binaries are problematic
        2. Nested hierarchies result in unexpected duplications
        3. Nested hierarchies result in long paths (windows issue).
    

2, and 3 will be resolved with NPM3, but that brings some interesting breaking
changes... beyond that is the fact that npm versions are generally pinned to
node/iojs version for most.

1\. I'm hoping to see a consistent implementation for tagging modules with
binary dependencies, or build dependencies so that there is a build platform
in place for at least common targets. Who knows if this will ever happen, and
likely will be tied to paid accounts, which isn't exactly a bad thing here.

------
joewalnes
A very tiny shell script: [https://github.com/joewalnes/go-
getter](https://github.com/joewalnes/go-getter)

Like go get, but you specify exact versions, including transitive
dependencies. One GOPATH per project. And most importantly, you don't have to
use go get to install the tool in the first place.

------
howeyc
Makefiles, build: docker build

Ugh, more Linux-Only. Don't follow this please.

(This isn't necessarily aimed only at blog post, but guys and gals there are
other OSs supported by Go than Linux/OSX. Most of the time just having "go
get" work is enough. Trust me, I've done it a lot as an outlier. I use FreeBSD
and Windows.)

What's wrong with using something like party[1], or nut[2]? Or just vendoring
in a way that go build still works. Seriously, what's wrong with more source
files in your project's source repository?

This way I can go get your command (package main).

I dunno, maybe I'm just grumpy.

[1] [https://github.com/jingweno/nut](https://github.com/jingweno/nut) [2]
[https://github.com/mjibson/party](https://github.com/mjibson/party)

~~~
vhost-
What? GNU make works everywhere. I've used it on Window, OS X, every Linux
distro I've touched and all the BSDs

~~~
howeyc
Yes, I know about GNU make. The point is not GNU make, the point is needing to
find out how to install third-party program on my OS of choice to build an app
that should be as easy as "go build" because of the Programming Language used.

Maybe once that new fancy OneGet is on Windows (and maybe even becomes as good
as brew looks on OSX, which I've never used) it will no longer bother me.

------
robfig
At work we have a monolithic repo containing Java, Go, Python, etc, organized
like this:

    
    
      Java //src/com/yext/..     
      Go   //gocode/src/yext/...
    

GOPATH is set //gocode. .gitignore allows us to track only our code under the
GOPATH using this snippet:

    
    
      /gocode/bin/
      /gocode/pkg/
      /gocode/src/*/
      !/gocode/src/yext/
    

We use glock[1] to sync dependencies across the team, without having to check
them in.

The whole thing works very well at 50 developers and ~100 dependencies.

[1] [https://github.com/robfig/glock](https://github.com/robfig/glock)

------
stock_toaster
Seems like the author is suggesting something very similar to what [gpm][1][2]
provides. I use gpm for a few projects, along with a makefile that creates a
GOPATH at $(pwd)/_build, and it works fairly well.

[1]: [http://java.dzone.com/articles/why-gpm-right-go-
package](http://java.dzone.com/articles/why-gpm-right-go-package)

[2]: [https://github.com/pote/gpm](https://github.com/pote/gpm)

~~~
0x74696d
Yeah, that plus gvp. But if you dig into both they're just shell scripts under
the hood (nice ones, though!) and given that I've got a makefile or shell
script to build the container, run tests, etc. then adding a third-party tool
is just one tiny bit more overhead.

------
skybrian
"go get" works for downloading open source code to check into a monorepo [1].
This is how Google does it. But most organizations use git, which doesn't
scale enough to support monorepos very well, and it's awkward if you're just
using git.

[1] [http://blog.rocketpoweredjetpants.com/2015/04/monorepo-
one-s...](http://blog.rocketpoweredjetpants.com/2015/04/monorepo-one-source-
code-repository-to.html)

------
dmitris
would not it be easier and cleaner to use the Android's REPO tool [1] to pin
down the versions and layout your workspace? At least the REPO allows you to
write down the dependency information in a declarative manner instead of
Makefile scripting snippets.

[1] [https://source.android.com/source/using-
repo.html](https://source.android.com/source/using-repo.html)

------
vkjv
I haven't written much go, so I'm just riffing here, but could you do
something similar to what the author is suggesting with git submodules?

~~~
sanderjd
I think this is a perfect use case for `git subtree`[0]. It actually pulls
everything in-repo, but can squash their history and do some other
conveniences. I've always been surprised at its relative obscurity – maybe
because of a terminology conflict with "subtree merges" – because it's so
useful. I haven't actually used it for go dependencies, but it seems like it
would be a good fit.

[0]:
[https://github.com/git/git/blob/master/contrib/subtree/git-s...](https://github.com/git/git/blob/master/contrib/subtree/git-
subtree.txt)

------
thelinuxkid
I did something similar to this a while back. It's just a few lines of bash.
[https://gist.github.com/thelinuxkid/c57ddaf9ddbdce2f30b9](https://gist.github.com/thelinuxkid/c57ddaf9ddbdce2f30b9)

------
lobster_johnson
I really like the OP's structure and makefile, but it looks like a pain to
work against Golang's grain here, even though the methodology makes a lot more
sense to me.

Maybe I have misunderstood the documentation, but what I truly, truly don't
get about Go's $GOPATH is that it wants (1) my project directory to have some
kind of canonical path, and (2) to pollute my project directory with
dependencies. I have done a bunch of Go development and I _still_ don't get
it.

So for example, I have myproject, which I naturally want to organize in this
way:

    
    
        $HOME
          Projects
            foo
              .git
              src
                main.go
                stuff.go
              assets
                photo.png
                something.xml
              scripts
                setup_database.sh
              config
                config.sample.json
              Dockerfile
              Makefile
    

Go wants me to throw this away and structure it like this:

    
    
        $HOME
          go
            bin
              [...]
            src
              github.com
                BurntSushi
                  toml
                    [...]
                zenazn
                  goji
                    [...]
                myaccount
                  myproject
                    .git
                    main.go
                    stuff.go
                    assets
                      photo.png
                      something.xml
                    scripts
                      setup_database.sh
                    config
                      config.sample.json
                    Dockerfile
                    Makefile
    

I'm supposed to work within this jungle of dependencies and generated files,
in the middle of which sits my project. At least Java, for all its many
faults, has the good sense to let you store dependencies nested as JAR files
in a subfolder, as opposed to turning your own project into a dependency.

I have tried fixing this by symlinking my project into $GOPATH/src/whatever,
but Go doesn't let me to run "go" commands on things outside the $GOPATH,
which makes this rather painful to do in a shell.

Adding to the general confusion, Go doesn't manage canonical package paths, so
github.com/zenazn/goji/main.go is in the package "goji" and
github.com/zenazn/goji/graceful/[asterisk].go are in the package "graceful",
but those package names only mean something to the importer. When "goji" wants
to use the package "graceful", it imports "github.com/zenazn/goji/graceful",
which, being a full URL, _must_ be resolved from a root folder in GOPATH/src.

At least Java (for all its... etc.) had the good sense to make dotted package
names mirror the folder structure of a project, not URL, thus being internally
consistent. Go wants everything to live in the same sea of things. Vendoring
is just half the problem.

~~~
wallyhs
Instead of throwing away your project structure, try creating a "go"
subdirectory in your project and setting GOPATH to point to it. Yes, you need
a separate GOPATH for each project, but it's no worse than needing a separate
Python virtualenv for each project.

~~~
lobster_johnson
That's what I'm doing these days. But it's painful.

It means I have to run full commands such as "go install <my package>" or "go
get <some package>". To install all dependencies, I have to do "go get <my
package>".

If I want to work on a project that depends on other projects that haven't
been pushed to master yet, I have to clone those projects and symlink them in
manually. It's pretty painful.

Working with certain tools that assume a certain directory structure (protoc,
for example, which computes paths relative to your Go root) is also painful.

"go build ." does work.

"No worse than virtualenv" isn't exactly a rousing endorsement. It's one of
the poorer package-management systems out there. Go should be as simple and
easy as Bundler. There's no excuse these days, I think.

~~~
wallyhs
You omitted that from your comment, but I agree. I was just pointing out the
option, because it didn't sound as if you had tried it.

I did not mean to endorse anything.

~~~
lobster_johnson
Yeah, sorry. I have been trying different things. Not happy. But at least it
seems that I'm not alone in being dissatisfied. It's surprising because Go
does a lot of things right, and disappointing because it's such a brake on
productivity.

------
jittuu
I believe it is in their radar. Go 1.6+ :)

[https://groups.google.com/forum/#!topic/golang-
dev/nMWoEAG55...](https://groups.google.com/forum/#!topic/golang-
dev/nMWoEAG55v8%5B1-25%5D)

------
kristianp
Mods, please rename to "Go Get Considered Harmful", as per the article's
title.

------
Walkman

        >>> ('{:c}'* 3).format(0x74, 0x69, 0x6d)
        'tim'

------
0x74696d
Author here. Happy to answer any questions about our setup.

------
sshb
We should have python's virtualenv analogue in Go.

~~~
namecast
I've been using Godeps, which feels like a close equivalent - have a gander
and see if it's useful:

[https://github.com/tools/godep](https://github.com/tools/godep)

Semver support was what got me using godep and I haven't looked back since (at
least for larger projects).

