
Using Go Modules - spacey
https://blog.golang.org/using-go-modules
======
ilovecaching
I've been using Go modules, and I really like them. It's obvious they really
paid a lot of attention to backwards compatibility. Being able to vendor using
mod is really awesome, as well as see all your transitive dependancies
automatically. It's very goish to focus on minimizing your dependancies and
actually think about what you're pulling in. I'm not yet sure how I feel about
the v2+ stuff. Having a separate module for breaking changes is something I
think I'll have to try more to form an opinion on, but I dislike the idea of
embedding the version in the name instead of using the mod file.

I do think it's kind of odd that it decides to put packages in GOPATH, but
disallows you from coexisting go mod projects in GOPATH. I organize all my
code for all my languages using the go repo as dir. I've had to maintain a
separate tree for go mod which is not ideal.

After using Go dep and mod, I felt that dep was the super straightforward and
obvious way to do it. It's how I would have done it. Go mod is much more
Goish, it's opinionated and based on a philosophy that fits with the language.
It Gives me hope that if they do add generics they will make them uniquely Go
as well.

~~~
jrockway
It does not put modules in GOPATH, at least not where you'd normally find
them. Pre-modules github.com/foo/bar would be in
$GOPATH/src/github.com/foo/bar. Now they are in $GOPATH/pkg/mod/... (... being
based on the URL with some handling for versions and characters that don't
work well in filesystems, I don't know the exact details).

The main problems I've had with go modules are fast-and-loose upstreams that
happily rewrite history. go mod detects this and stops the world, and you have
to manually edit go.sum.

I also feel like "go get github.com/foo/bar" doesn't always get me the latest
version. You see that there was a commit 1 hour ago, but then "go get" adds a
version like 20181130-93874837 to your dependencies. I assume they know what
they're doing? But I'm probably wrong.

~~~
SamWhited
> The main problems I've had with go modules are fast-and-loose upstreams that
> happily rewrite history. go mod detects this and stops the world, and you
> have to manually edit go.sum.

This is really irritating. I have the same problem with TLS: every time a
malicious actor tries to manipulate my connection and steal my cookies I get
an error page and can't do online banking anymore!

Sarcasm aside: this is not a problem with Go modules. This is a problem with
the libraries you're using and you should avoid those projects until they get
their act together.

~~~
kasey_junk
Internal libraries are a huge pain to build now. I don’t need the rigor of
semantic versioning for this, it just gets in the way. My act is together, I
have tests that confirm my libraries work, I have a idempotent versioning
system that is automatic (git sha) and if something gets merged to master that
is my stamp of approval that it works.

Between the versioning behavior and trying to figure out how to document
transitive dependencies correctly go mod has made my life as a maintainer
harder.

~~~
knownunown
It sounds like the replace syntax might help here. It allows you to forgo
semver+version control and depend on modules on your local disk.
[https://github.com/golang/go/wiki/Modules#can-i-work-
entirel...](https://github.com/golang/go/wiki/Modules#can-i-work-entirely-
outside-of-vcs-on-my-local-filesystem)

------
joshklein
Making sure I understand this: I’m some independent developer working in many
languages. I like every project I work on to be inside ~/work/ within a subdir
I name based on the project.

I used to be annoyed that I had to put every go project into a dir 7-8 layers
below that e.g ~/work/go/src/github.com/joshklein/project/cmd/hello_world.go,
but now I can have ~/work/go_hello/src/main.go. Right?

I understand this isn’t the main point, but frankly it’s the thing I care the
most about.

~~~
ilovecaching
I actually organize all my projects using the go way, there's nothing that
stops you from coexisting Haskell in the Go tree. The advantage is that if you
are working on something like the kernel, you can maintain separate branches
easily.

That said, mod allows you to put your project anywhere but in the GOPATH, so
the answer to your question is yes. The path is virtualized in the mod file.

~~~
Waterluvian
It awfully breaks when you run into multiple systems that want things done
their way. Like Go and ROS workspaces.

------
mikepurvis
I'm not an active golang user, but something which gives me pause here is the
insistence on a strict 3-number semver.

For a commercial entity shipping software which may include upstream
components, it's important to have options for maintaining (and versioning)
in-house forks of upstream components. This becomes a problem when you fork
v1.2.3 from upstream and want to internally release your fixes, but now your
internal 1.2.4 and upstream's 1.2.4 both have the same number but diverge in
content.

I like the Debian solution to this, which permits freeform textual suffixes,
so that in the hypothetical scenario above, you can release v1.2.3bigco1,
v1.2.3bigco2, with the final number indicating the version of the patchset
being applied onto the upstream release; then it's also clear what to do when
you rebase your fork because you've maintained the integrity of the upstream
version.

~~~
LukeShu
_> I like the Debian solution to this, which permits freeform textual
suffixes, so that in the hypothetical scenario above, you can release
v1.2.3bigco1, v1.2.3bigco2, ..._

Go also allows freeform textual suffixes, it just needs a `+` in there:
v1.2.3+bigco1, v1.2.3+bigco2.

~~~
felixfbecker
Oh wow, so they are not following semver? In semver everything after + is a
build tag that does not "modify" the version (i.e. 1.2.3+a and 1.2.3+b are
considered to both be the same version, just e.g. built at different times).

~~~
busser
The blog post links to the Semantic Versioning spec, specifically item 9.

> A pre-release version MAY be denoted by appending a hyphen and a series of
> dot separated identifiers immediately following the patch version.
> Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-].
> Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading
> zeroes. Pre-release versions have a lower precedence than the associated
> normal version. A pre-release version indicates that the version is unstable
> and might not satisfy the intended compatibility requirements as denoted by
> its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1,
> 1.0.0-0.3.7, 1.0.0-x.7.z.92.

Item 10 explains how + can be used to add a build tag.

> Build metadata MAY be denoted by appending a plus sign and a series of dot
> separated identifiers immediately following the patch or pre-release
> version. Identifiers MUST comprise only ASCII alphanumerics and hyphen
> [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata SHOULD be
> ignored when determining version precedence. Thus two versions that differ
> only in the build metadata, have the same precedence. Examples:
> 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85.

Instead of using v1.2.3bigco1 or v1.2.3+bigco1, we should use v1.2.3-bigco1.
If we need to add a build tag we can use v1.2.3-bigco1+001.

------
Animats
_" the go command automatically looks up the module containing that package"_

Looks up where? GOPATH? The tree below the current file? The current
directory? What's the "current module", anyway? Where is that stored?

 _" go: downloading rsc.io/sampler v1.3.0"_

Downloading from where? Github? From what repository? The page for "go get"
now talks about "module-aware" mode, but doesn't make it clear how that works.

This is the usual headache with dependency systems. There's some specific
directory layout required, and you have to know what it is. This article needs
to be more explicit about that. Otherwise you end up depending on lore and
monkey copying.

~~~
LukeShu
_> What's the "current module", anyway? Where is that stored?_

The article uses "current module" like you might use "current git repository";
it's the module containing the directory that you are currently `cd`ed to.
Just as git identifies the repo by looking for `.git` in successive parent
directories, go identifies the module by looking for `go.mod` in successive
parent directories.

 _> > "go: downloading rsc.io/sampler v1.3.0"_

 _> Downloading from where? Github? From what repository?_

From the same place that non-module-aware `go get` downloads it. The only
difference on the "remote side" of that from old `go get` is that it grabs the
`v1.3.0` git tag, instead of git HEAD; if you don't specify a version it grabs
the most recent `vSEMVER` git tag instead of git HEAD.

 _> > "the go command automatically looks up the module containing that
package"_

 _> Looks up where? GOPATH? The tree below the current file? The current
directory?_

Looks it up on the network, à la `go get`. Exceptions:

\- The `replace` directive in your `go.mod` can manually override where it
looks up a specific package.

\- Telling Go `-mod=vendor` will have it look up all depended-upon modules in
`vendor/`, rather than via the normal mechanisms; you can create the `vendor/`
directory with `go mod vendor`.

It doesn't use the network every time; a module cache lives in
`${GOPATH:-${HOME}/go}/pkg/mod/`; but you don't need to know that any more
than you need to know that a build cache lives in
`${GOCACHE:-${HOME}/.cache/go-build}`.

 _> There's some specific directory layout required, and you have to know what
it is._

Not really, the only requirement is that you have a `go.mod` file that
identifies a package name for the directory that it's in. So that if I have
github.com/lukeshu/foo, it just needs a `go.mod` saying

    
    
        module github.com/lukeshu/foo
    

instead of having the requirement that it be checked out to
`$GOPATH/src/github.com/lukeshu/foo`. Beyond having the `go.mod` file, there
aren't really any structure requirements.

 _> This article needs to be more explicit about that._

Does it, though? It's a blog-post, not the "normal" documentation. The full
docs _are_ much more explicit and nitty-gritty than a high-level blog post.

~~~
Manishearth
As far as I can tell, rsc.io/sampler isn't a git repository, so it's not doing
the same stuff `go get` does. If I try to clone it I just get hit by the HTTP
redirect to the docs for rsc.io/sampler.

I may be missing something here (it's been quite a while since I did serious
Go stuff so `go get` may have changed too)

> Does it, though? It's a blog-post, not the "normal" documentation. The full
> docs are much more explicit and nitty-gritty than a high-level blog post.

It's a bit weird to have a blog post about the new module system with nothing
about how modules can be published, a pretty common task.

~~~
LukeShu
_> As far as I can tell, rsc.io/sampler isn't a git repository, so it's not
doing the same stuff `go get` does. If I try to clone it I just get hit by the
HTTP redirect to the docs for rsc.io/sampler._

Why would you try to `git clone` it directly instead of trying to `go get` it?

    
    
        $ GO111MODULE=off go get -v rsc.io/sampler
        Fetching https://rsc.io/sampler?go-get=1
        Parsing meta tags from https://rsc.io/sampler?go-get=1 (status code 200)
        get "rsc.io/sampler": found meta tag get.metaImport{Prefix:"rsc.io/sampler", VCS:"git", RepoRoot:"https://github.com/rsc/sampler"} at https://rsc.io/sampler?go-get=1
        rsc.io/sampler (download)
        created GOPATH=/home/lukeshu/tmp-go; see 'go help gopath'
        Fetching https://golang.org/x/text/language?go-get=1
        Parsing meta tags from https://golang.org/x/text/language?go-get=1 (status code 200)
        get "golang.org/x/text/language": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/language?go-get=1
        get "golang.org/x/text/language": verifying non-authoritative meta tag
        Fetching https://golang.org/x/text?go-get=1
        Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
        golang.org/x/text (download)
    

It's been that way since at least 1.2 (I'm pretty sure since 1.0, but I
haven't verified that memory). The process that `go get` follows is described
by `go help importpath`.

The TL;DR is that it fetches `[https://${pkgpath}?go-
get=1`](https://${pkgpath}?go-get=1`), and looks for `<meta name="go-import"
content="...">` to tell it where to `git clone` from (with some hard-coded
hacks for common websites that don't support that, like github.com).

 _> It's a bit weird to have a blog post about the new module system with
nothing about how modules can be published, a pretty common task._

Because that hasn't changed. You still publish packages the same way you
always have. The only difference in how you publish is that now "vSEMVER" git
tags mean something.

~~~
Manishearth
Ah, I wasn't aware that the meta tag existed, everything about publishing
crates seemed to point to using github (or a self-hosted repository of various
supported VCSes)

Thanks.

The example using a less-commonly-known package distribution method is
probably causing the confusion in Animats' original comment.

------
dimgl
I've been using Go modules for a bit and I really like it. I can easily set up
local dependencies using the `replace` keyword as well. For instance, all of
the apps in `cmd` which need access to common packages are their own modules,
and then I just add:

    
    
      replace pkg => ../../pkg
    

And I can refer to everything inside of `pkg` (`pkg` is its own module as
well).

------
sethammons
The meat:

> go mod init creates a new module, initializing the go.mod file that
> describes it.

> go build, go test, and other package-building commands add new dependencies
> to go.mod as needed.

> go list -m all prints the current module’s dependencies.

> go get changes the required version of a dependency (or adds a new
> dependency).

> go mod tidy removes unused dependencies.

And you can use some sugar to declare exact versions you want.

------
fourseventy
The main thing that annoys me about Go modules is that it broke so much
tooling. I still can't get GoCode to work properly, and there are 4 or 5
different forks of the repo all trying to add module support, its a mess.

~~~
throwaway8879
Gocode works in vim but for some reason won't do autocomplete in spacemac's
go-mode. I haven't found the time to look for a fix yet. There's a fork of
Gocode that fixes it, but makes vim's autocomplete very very slow. As someone
who switches between vim and emacs I wish this mess gets sorted out soon!

~~~
farslan
HEAD of vim-go has now gopls support, which has excellent support for Go
modules for code completion and jumpt to definition. Give it a try.

~~~
throwaway8879
Oh, I'm already using vim-go, but didn't know it does code completion too.
Thanks for writing such a great plugin!

------
antwerpen
Google API's latest semantic version is behind the doc site
(godoc.org/google.golang.org/api). Is this standard practice in Go?

I was coincidentally converting my Go project to use Go modules yesterday. I
depended on a Google API which was originally retrieved via 'Go get',
corresponded to docs and worked fine as this pulled from HEAD. `go mod` did
not work out of the box, as it required the latest semantic version (v.0.2.0)
of my this Google API import. This version, however, is behind documentation
and broke my code.

I understand I can require a specific commit in the go.mod file, but the
strings for specific commit seem cryptic. Where can I look up the version hash
that matches the doc site?

~~~
antwerpen
Ah. The trick is to substitute in the go.mod file's require block.:

s/$SEMANTIC_VERSION_BRANCH/$BRANCH

In this case, $BRANCH == 'master'. 'go build' will resolve 'master' to a
specific commit hash version.

~~~
chrisbroadfoot
you can also `go get google.golang.org/api@master`

------
minieggs
I've returned to university recently. We're learning Make in one of my classes
at the moment ("learning" Make seems so weird to me). When going over
dependency graphs I couldn't stop thinking of how awesome `go mod tidy` is.

------
presscast
I haven't been using Go modules, to the point where I only have a vague notion
of what they do.

I haven't felt the need for them.

At what point do they become necessary? Which pain-points should I be on the
lookout for?

~~~
yiyus
Probably at the point new versions of your dependencies break your code. If
you have no external dependencies, you should be fine.

It also gives you the freedom to have your projects wherever you want.

------
nicpottier
Be warned that if you like all the tooling VSCode provides for Go then that is
still not available if you are using go modules. They are working on it and
there are some rough betas out there for some limited functionality but simple
things like renaming variables etc, are still not available.

Realize this is a tooling issue and not necessarily a language one, but do
feel like they should be mentioning this. It makes working with modules pretty
painful. (we are doing it but man do I miss simple refactors / usages etc..)

~~~
le_didil
I agree, I tried go modules a couple of months ago but the VSCode tools were
much slower and cpu usage increased when using go mod extensions. I reverted
back to not using go modules at the time, will try again soon to see if it
improved. I still like go modules.

------
fiatjaf
All I wanted was a way to safely and easily cleanup my $GOPATH. I don't have a
lot disk space and $GOPATH takes up most of it.

This thing apparently doesn't solve my problem. `go mod tidy` simply removes a
dependency from a module, but that dependency is still cached in a big arcane
directory at $GOPATH/src/mod. Why?

I think we all should be using something like Nix for dependency management
that solves all problems, but that is so hard to setup!

~~~
jerf
Not having a lot of disk space is a minority case for developers now. I
wouldn't hold your breath for the Go authors to address it.

If you have a disk space problem, Nix would seem to be the _opposite_ of the
solution to that. One of the ways Nix does its magic is to chew through disk
space a lot more freely than most distros do.

~~~
fiatjaf
Also, there are a lot of people who think disk space is an issue. See
[https://pnpm.js.org/](https://pnpm.js.org/) for example.

(Ironically, pnpm doesn't have an auto cleanup feature.)

~~~
jerf
npm was especially broken with disk space, above and beyond what Go has ever
had, so their problem was much worse.

And my point is not that disk space is never an issue. That's why I said it
was a _minority_ issue, not "not an issue". My point is more than Go is not a
language about addressing every fiddly minority's issues. It's definitely
about the 20% that does 80% of the work. So waiting for a language lead by a
philosophy like that to address a disk space issue is probably not a good
plan.

To be a bit more constructive, I'd observe I've had great experiences with
cross-compiling. On the chance you're disk-space limited because you're
developing right on a target resource-constrained device, you may be able to
move your development to a more powerful system, even of a different
architecture, and cross-compile fairly easily. I often have a workflow where I
just "go build blahblahblah && rsync blahblahblah target:blahblahblah && ssh
target blahblahblah" and I just press up & enter on a shell when I want to
push & test the code. As long as you've got half-decent bandwidth to the
target device, it's fine. There may be a couple of other buttons you may want
to push to speed that up, because IIRC when cross-compiling it'll end up
building the entire app, and you'll want to pre-compile things for the new
arch.

------
aw4y
question: what happens if you wanna organize your software in modules? so for
example, a main module (package main) and a secondary module (core). Do you
init both as separate modules? and then you use the second as dependency on
the first one? do they have to be actually published somewhere to be
accessible in this way?

~~~
skybrian
You could use an unpublished version of "core" using the "replace" directive,
but this might not be a good idea. If you have two modules that are so closely
tied together that they always need to be released simultaneously, it's
probably better to make it a single module with multiple packages in it.

~~~
aw4y
thanks!

------
skybrian
There are at least two bugs in this tutorial:

\- There's a duplicated section starting with: "Note that our module now
depends on both rsc.io/quote and rsc.io/quote/v3:"

\- In the example of upgrading a major version, Hello was renamed to HelloV3,
but the caller isn't renamed. It should be "quoteV3.HelloV3".

------
gfs
In the "Upgrading dependencies" section when they use "go get", will the
command be local to that module? I am used to "go get" working with the idea
of a $GOPATH. I almost wish they introduced another sub-command to update a
dependency.

------
tschellenbach
not ready yet for production, most of the cli tools such as errcheck dont
work.
[https://github.com/golang/go/issues/24661](https://github.com/golang/go/issues/24661)

other than that it works like a charm, here's a tutorial i wrote:
[https://getstream.io/blog/go-1-11-rocket-
tutorial/](https://getstream.io/blog/go-1-11-rocket-tutorial/)

~~~
LukeShu
Do note that you can use errcheck with modules if you call it through
golangci-lint.

------
mmgutz
How do you reference local go modules that are under development? The
equivalent in node is `npm link`.

~~~
_mway
with a `replace` entry. e.g.

    
    
        replace (
            github.com/repo/module/path vX.X.X => /path/to/github.com/repo/module/path
        )

------
mirceal
Every mainstream programming language that came out in the last 20 years has
already solved this. Java, Javascript, Ruby, Python, Rust, you name it.

I am baffled to why Go has not figured this out and are presenting recent
developments as some kind of breakthroughs - they are not.

To me, and I'm not trying to be disrespectful here, because of the
shortcomings it has, Go is in the experimental/keep an eye on bucket. It does
some things really, really well but it's far from the panacea most developers
that use it think it is.

~~~
weberc2
Go's modules are quite a lot nicer than Java's or Python's dependency
management / build tools. JavaScript only recently got its act together. Can't
speak to Ruby, but Rust is the only one that got it right the first time.
Dependency management is only recently a "solved problem".

> Go me, and I'm not trying to be disrespectful here, because of the
> shortcomings it has, Go is in the experimental/keep an eye on bucket. It
> does some things really, really well but it's far from the panacea most
> developers that use it think it is.

Go is definitely a production-ready language, and indeed it powers much of the
most important software that has been written since it went 1.0 (particularly
in the Cloud and DevOps spaces). It's not perfect, but it's really, really
good and improving steadily.

~~~
geodel
> Rust is the only one that got it right the first time.

Not really the first time. They made attempts before. Dumped it along with
person who created earlier solution and then developed current solution.

To their credit they identified early that an official solution is essential.

~~~
mirceal
> To their credit they identified early that an official solution is
> essential.

this made me smile.

