
How to start a Go project in 2018 - boyter
https://boyter.org/posts/how-to-start-go-project-2018/
======
zimbatm
It's distasteful of Go to impose a filesystem layout.

Any other language that I work with understands that dependencies should be
self-contained within the project folder. Each project has it's own set of
dependencies that have been tested to work together. The user can select where
to checkout the project, or even have multiple copies of the same project
lying around.

To achieve the same thing with Go, one has to set a different GOPATH per
project, and then checkout the project deep into that root. This is not
convenient as a developer and as a software packager. Or give in and having to
resolve dependencies that work with all the current projects. Given that Go
doesn't really do package versioning, finding the right set of commits that
all work together is a exponential nightmare.

Now that since Go 1.6 each project can have its own vendor/ folder, GOPATH
shouldn't be required. I should be able to checkout the project where I want
and then have the tools look into the vendor/ folder to resolve any
dependencies. Please change it that way. The reason govendor and all these
tools are complicated is because of this GOPATH madness. Synching dependencies
between GOPATH and vendor/ should only be a problem for Google, not the rest
of us.

~~~
strkek
I made[1] a zsh hook some time ago that allowed you to have per-project
GOPATH.

At the root of the project I just have to do `echo
'github.com/username/package' > .gopkg` and after that, for everything I do
under that directory tree, GOPATH will be automatically set to something like
`export GOPATH="$(dirname .gopkg)/.gopath"`.

I don't use zsh anymore though.

[1]:
[https://gist.github.com/strkek/294c2f5e6fe94b8303b5121266b27...](https://gist.github.com/strkek/294c2f5e6fe94b8303b5121266b27a22)

~~~
zimbatm
nice, you could do something similar using direnv[1]

[1]: [https://direnv.net/](https://direnv.net/)

------
simula67
> Go dependencies are a little odd the first time you run into them. I suspect
> this is because Google runs a mono-repo and as such the decisions around it
> were made with that in mind.

I think the oddness is around the directory structure. It may also be because
Go was developed by UNIX old hands. When UNIX was designed, it was not just
for ordinary users, but also developers. When developing in UNIX, you can use
all of UNIX as an IDE ( grep, find etc ). Most Unices have a folder called
/usr/src [1] where sources are stored. When you want to build a package, you
cd into it ( say 'cd /usr/src/make') and then say 'make && make install' and
it builds and installs the packages to your /usr/bin.

I am assuming, Google's internal monorepo, which is derived from Perforce [2]
encourages you to keep the /usr/src under source control. Since Piper is not
released to the public and most people now use git, GOPATH was probably
conceived as a way to relocate '/usr/src/' to another location. This is
probably why 'go install' ( which probably mimics 'make install' ) installs
everything at 'GOPATH/bin'. If this is true, I think we have a mental model on
Go's directory structure.

Hopefully someone can tell me if I am off base.

[1]
[https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)

[2] [https://cacm.acm.org/magazines/2016/7/204032-why-google-
stor...](https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-
billions-of-lines-of-code-in-a-single-repository/fulltext)

~~~
jrockway
To be fair, go code at Google doesn't use the standard directory structure. I
had all my go code in one directory and which files were part of which package
was all defined in a build file.

What exists in the public world of go was simply what the go team wanted for
go and not a compromise to fit in with other systems at Google. I think the
fact that you end up with what looks like a monolithic repository is just a
coincidence.

In the end ~/go doesn't look a lot different than the @INC path that points to
something in your home directory, or /usr/include, /usr/lib, and /usr/bin in a
pure C system. It's just the fact that you also put your own code in the same
directory as third-party packages that confuses people... but in the end,
someone else will consider your package a third-party package someday... so I
don't think there's much value in treating it any other way.

~~~
strkek
> In the end ~/go doesn't look a lot different than the @INC path that points
> to something in your home directory, or /usr/include, /usr/lib, and /usr/bin
> in a pure C system.

Well, yes and no, and that's the only thing I don't like about GOPATH: that
your source code is not in GOPATH, but in `$GOPATH/src`.

I'm not against GOPATH per-se because I'm already used to other PATHs; I'm
just against it not behaving like those other common PATH variables, like
PATH, LD_LIBRARY_PATH, PYTHONPATH[1], etc. GOPATH introduces intermediary
directories (src, pkg, bin), instead of going straight to the point and being
only for source code.

I mean, there's `GOPATH` and also `GOBIN` which is just `$GOPATH/bin`. IMO it
would make much more sense for GOPATH to be _only_ source code, GOBIN to be
only executables, and something like GOPKG for the current `$GOPATH/pkg`.

[1]: Yes I know venv is a thing, I'm just mentioning PYTHONPATH to illustrate
my point.

~~~
kchr
Fully agree. Having the option to split these three domains would make it a
lot easier to have basically any directory structure for your projects and how
it can be distibuted.

------
aodin
If you're using Go with Docker 17.05 or higher, I recommend a multi-stage
Docker build. Official documentation here:
[https://docs.docker.com/develop/develop-images/multistage-
bu...](https://docs.docker.com/develop/develop-images/multistage-build/#use-
multi-stage-builds)

The default Go Docker images include everything needed to compile Go. But once
you have a binary, you really only need an image capable of running your
binary. I've seen images sizes reduced by over 95%.

Revising the example in the article:

    
    
        FROM golang:1.10
        COPY . /go/src/bitbucket.code.company-name.com.au/scm/code
        WORKDIR /go/src/bitbucket.code.company-name.com.au/scm/code/
        RUN CGO_ENABLED=0 go build main.go
    
        FROM alpine:3.7
        RUN apk add --no-cache ca-certificates
        COPY --from=0 /go/src/bitbucket.code.company-name.com.au/scm/code/main .
        CMD ["./main"]
    

The `CGO_ENABLED=0` and `apk` are oddities of Alpine Linux specifically. The
image you choose to run your binary may not need these.

~~~
nemothekid
I've been using `scratch` final builds .-

    
    
        FROM golang:1.10
        WORKDIR /go/src/bitbucket.code.company-name.com.au/scm/code
        COPY ./ ./
        RUN CGO_ENABLED=0 go build -a -installsuffix cgo -o /dist/main .
    
        FROM scratch
        COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
        COPY --from=0 /go/src/bitbucket.code.company-name.com.au/scm/code/dist/main /
        ENTRYPOINT ["/main"]
    

It requires a bit more "magic", but is a smaller build at the end.

~~~
reacharavindh
Serious question: Why do you need Docker for a Go project? Cant you build it,
and copy the binaries over to the host you want to run it on?

~~~
d4l3k
There's definitely use cases for Docker with Go that mostly revolve around
running untrusted code or wanting to be able to set tight restrictions on CPU
usage, memory etc. Though you could also just do the latter via cgroups.
Another use case might be if you have extra files that need to be deployed
with the binary as well.

I wrote up some documentation about this for my use case just in case I ever
have to go back and change it.

You can see it here: [https://fn.lc/post/docker-
scratch/](https://fn.lc/post/docker-scratch/)

~~~
dijit
>There's definitely use cases for Docker with Go that mostly revolve around
running untrusted code.

Please don't run untrusted code in Docker, it's not designed for security.

[https://security.stackexchange.com/questions/107850/docker-a...](https://security.stackexchange.com/questions/107850/docker-
as-a-sandbox-for-untrusted-code/108013)

~~~
Zekio
Docker containers can run in other places where they are secure, IIRC then
Joyent has way to do this in their cloud

~~~
dijit
You havent refuted my point. People take a stance that docker can run
untrusted code without actually looking at what they have to do for that to be
true.

~~~
d4l3k
I haven't taken the stance that all docker containers can run untrusted code,
but I've certainly done my best to harden my docker containers as much as
possible. If you took a look at that write up I cover it.

I run with the flags: --net=none --cap-drop=all --cpus=1 --read-only
--tmpfs=/tmp:rw,size=1g,mode=1777,noexec

This gives it no way to communicate with the outside world and drops all
capabilities so it's not allowed to interact with the kernel at all. Setting
CPU limit to be 1 also prevents DOS attacks internally.

I also run the process under the nobody user which entirely avoids the
"container root is system root" issue.

I'm also only sort of running "untrusted" code. I'm running tensorflow models
which can do arbitrary computation but are more secure than just running raw
code.

------
KyeRussell
It feels like a slight failure that things like this are necessary, especially
for a new language.

As a Python developer, I am no stranger to slightly ridiculous packaging /
environment woes. However Go has 19 years on Python. I understand that they
are different languages and ostensibly serve different purposes, but it feels
like a shortcoming to me - based on the little Go experience I've had.

~~~
faitswulff
> As a Python developer, I am no stranger to slightly ridiculous packaging /
> environment woes.

Speaking of which, what is the best way to start a python project in 2018,
now? Python's version and dependency management is my least favorite aspect of
the language.

~~~
undecidabot
It's still a bit new, but I think pipenv [1] is the new standard now. No need
to directly work with pip or virtualenv anymore, and the equivalent
requirements file works more like npm/yarn.

[1] [https://github.com/pypa/pipenv](https://github.com/pypa/pipenv)

~~~
jerf
With all due respect to the Python community, and without intending this as an
insult per se, if this really is the "current recommendation", the Python
community is in no position to be criticizing the Go community on this point.
This is about the fourth answer I've heard in the past 10 years. I've been
programming off and on in Python for at least 15 years and I've never heard of
this tool.

Out of morbid curiosity, what's the current answer for "how to pack up a
Python program into a single executable/directory for Windows" now?

~~~
KyeRussell
Am I meant to justify all Python-related decisions because I choose to work
with it? Does my using Python (or even my membership in 'The Python
Community') disqualify me from critiquing on other languages and development
environments?

What does your comment add to the discussion other than snark and aggression?

~~~
jerf
A correction to people claiming that Python's obviously got a better solution
to dependency management than Go, a solution so "obvious" that a Python
programmer of 15+ years hasn't heard of it yet. (No, I'm not actively
programming in Python right now, but if it's so obvious and concrete I should
still have known about it. I was up-to-date on best practices as of a couple
of years ago!)

You aren't obligated to do anything; the question is open to anybody.

~~~
mixmastamyk
The original post says golang should be a bit better starting twenty years
later.

Pipenv has only existed a year or two, qed.

------
agnivade
Fair summary. I would like to add a couple of quick points that we are working
on.

* GOPATH is going to go away soon. Once vgo gets fully baked in to the language. Probably by 1.12 - [https://github.com/golang/go/issues/4719](https://github.com/golang/go/issues/4719).

* There is a one-line installer called "getgo" which installs Go. However, it needs further polish which is being worked on - [https://github.com/golang/go/issues/23381](https://github.com/golang/go/issues/23381), [https://github.com/golang/go/issues/21277](https://github.com/golang/go/issues/21277).

------
tyrankh
Couple of quick notes from my phone, as a googler gopher:

\- IIRC package management was historically left to the community to solve
for, rather than the go maintainers mandating how package mgmt should be done.
Many languages have followed this model. I've never heard of it having to do
anything with our mono repo model. The community never ended up standardizing
on a model; since then, the go team has endorsed godep as the official
experimental package manager, whose learnings were used to design vgo. Vgo is
very new, and still being iterated on; check it out though!

\- vgo removes the need for GOPATH

\- There is a one-line installer to set up your env. It's very useful and
simplifies a lot of what this blog talks about.

Sorry I don't have links - on phone at airport! All the above should be easily
googleable though! :)

~~~
imjasonmiller
If I may ask, how do you handle the use of other languages within a project
that uses Go? I am new to Go, so it might be a silly question.

I currently have a project that uses Node and React for the front-end and Go
for streaming and scraping large .xml + .json data due to its great standard
library.

With Node I can start a project wherever I would like — e.g.
D:/work/node_project. Go seems to be much more opinionated on that? I opted to
follow Go’s preferred directory structure and include npm related packages
there, but feel I am not following best practices. Ideally, I would like to
have one directory for my projects.

I will take a good look at vgo this afternoon. It removing the need for GOPATH
sounds interesting. Thank you.

~~~
nzoschke
I make the go server/package the repo and add a web/ subdirectory for my
static and dynamic JavaScript code, plus a little bit of Makefile glue.

Here is a boilerplate Go Lambda app with a Node Lambda function and a static
Vue app:

[https://github.com/nzoschke/gofaas](https://github.com/nzoschke/gofaas)

------
alexkavon
> For Windows it allows you to share the directory between the WSL and Windows

I've avoided changing the GOPATH by using the following set up:

C:/Users/<user>/go (Standard windows GOPATH)

Then symlink in WSL:

ln -s /mnt/c/Users/<user>/go ~/go

Works pretty well. Then you can install Go on Windows if you need it,
otherwise your dev environment is isolated and your Go code remains on Windows
if you remove/refresh WSL. Also if you use file history, the code will be
backed up this way in your Windows home as well.

------
lucidone
The biggest blocker in getting me to give Go a shot is it forcing a certain
directory structure on me, and the lack of Gemfile or package.json equivalent.

Has this been solved? I really want to use Go because of everything I've
heard, but I can't stand how $GOPATH forces a directory structure on me.

~~~
icholy
After using Go for a while, I use the GOPATH style layout for all my projects.
It just makes sense.

~~~
eknkc
Yeah I started doing this too. My own code folder is basically my github user
folder.

Keeps everything organized.

------
ironjunkie
Nice summary.

The biggest pain with Go is the dependency management. I initially started to
commit the whole vendor folder for each project, but lately I only commit the
Gopkg.lock and Gopkg.toml

Both those approaches bothered me over time and I'm still undecided about
what's the best way to do this. I know some projects only commit their
Gopkg.toml (and not lock) with some explicit dependencies.

~~~
pietroglyph
> I initially started to commit the whole vendor folder for each project, but
> lately I only commit the Gopkg.lock and Gopkg.toml

This really bothers me too; I appreciate Go's liberal application of
convention over configuration, but I feel like this part of dep is a break
from the simplicity afforded by the aforementioned principle. I often
appreciate this in other Go tooling (e.g. gofmt), and I would love if they
just made everyone do dep one way (I don't care which.)

------
viraptor
I'd add:

One you set it up, document exactly that setup in readme. If you use something
specific for dep management, document it. If not, document it. Document how to
run tests. Document everything...

At work I often run into issues with "random project X on GitHub" and do a bit
of drive-by fixes. But many times when I find some Go project, I have no idea
how to get from a fresh repo to running tests. And if that requires a lot of
time to figure out, you're getting a vague issue that I may not want to spend
time on, rather than a ready PR. Unfortunately this happens for go projects
much more than other languages in my experience.

~~~
Cthulhu_
I'd say this is true for all projects tbf; set up a readme, type what you need
to know and how to run the project in dev mode. Shouldn't take more than five
minutes to type out, or copy / pasted from another project.

------
nunez
It’s a good thing that Go is such a good language, because getting started
with it is downright PAINFUL compared to Ruby or Python.

~~~
mathw
It's so painful to get started that I'm not at all convinced the okay-ish
language experience is worth it. I feel like Go was on the verge of something
brilliant but managed to miss it by a couple of significant areas.

~~~
coldtea
Painful to get started in which way?

Is this about being used to dynamic programming and not groking types and
"pointers" in Go?

About the tooling?

About the lack of certain features like generics?

------
ospider
`dep` is actually a good tool and suppose to be the official tool. But out of
no reason, `vgo` came out and said to be deprecating `dep`. I don't really
understand how the Go team make decisions.

~~~
shabbyrobe
Go in general seems to be powered by a general principle of "Was it invented
here?", which undoubtedly stems from its plan9 roots as an upstart reinvention
of Unix. The software community tends to treat the "not invented here"
phenomenon as detrimental but I think in the case of Go, the willingness of
the team to reject the status quo has been a significant contributor to its
success and utility.

But for all the good this attitude brings (and I would definitely regard it as
a net benefit) there have been some pretty huge missteps too, and the Go
team's prioritisation of things like type aliases or the half-finished plugin
architecture over those missteps leads many to ask reasonable questions about
how those priorities are decided.

GOPATH is a long-acknowledged mistake and a lot of community effort and
consensus-building work has gone into finding a solution. vgo was a bit of an
ambush and has left a lot of people confused and bewildered, especially after
a huge amount of community goodwill and momentum was built up around dep. That
was a price Russ Cox decided to pay with vgo, which was his prerogative. Now
we just need to wait and see whether it was worth it, which will take some
time.

~~~
shabbyrobe
And in an astonishing piece of timing, the author of dep has posted an
analysis of vgo:
[https://news.ycombinator.com/item?id=17063724](https://news.ycombinator.com/item?id=17063724)

------
Sir_Cmpwn
I think this article is a little bit shortsighted. You should probably be
studying vgo in 2018 instead of using the old and soon-to-be deprecated way.
Of course, this being golang, I'm about to get a bunch of replies from people
who think they know better suggesting that vgo is terrible and describe their
stockholm-syndrome-like relationship with GOPATH.

vgo is the only reason I am willing to give Go the time of day in 2018.

------
dsymonds
> To search for anything about Go in your search engine of choice use the word
> golang rather than go when searching. For example to search for how to open
> a file I would search for golang open file.

This seems to be a recurring chestnut, but I would have expected the author to
actually test it. If I search for [go open file], 8 of the first 10 results on
Google show useful information.

------
brunoluiz
I was expecting a way to organise a golang folder structure for web apis
(2018), as there are some old tutorials out there

~~~
ovao
The overall approach to folder structure can be a personal thing, and doesn't
matter too much what you settle on. My own preference is for something like:

    
    
      github.com/<user>/<repo>
      |- main.go
      |  |- config/
      |  |- db/
      |  |- server/
      |  |- router/
      |  |- handler/
      |  |  |- get/
      |  |  |- put/
      etc.
    

In this setup, the router sub-package might import handler/get, handler/put,
and so on, returning a router, which can typically be passed as an
http.ServeMux to a Start or Run func in the server sub-package. main() winds
up doing the bulk of any initialization sub-packages need, but not much else.

At various points you'll probably find that sub-packages become general enough
that they can be easily broken out into their own repos and maintained
separately. This can wind up being a good goal to shoot for when designing the
APIs for each sub-package.

~~~
brunoluiz
But wouldn't be better to organize it in domains, together with clean
architecture? Because if the application have too many domains, it can get a
bit messy, doesn't it? (Although if a micro-service strategy is used this
shouldn't be a problem)

~~~
ovao
If your application requires multiple servers running multiple domains, you're
_probably_ better off with a microservice architecture.

------
zuzuleinen
I have to say, I love his minimal blog page. Loads fast and looks great :)

------
raydac
also it is possible to load SDK and tune all environment through maven (of
course maven and java should be installed) [https://github.com/raydac/mvn-
golang-example](https://github.com/raydac/mvn-golang-example)

------
zerr
Is import "www.geocities.com/lib" still a thing in Go?

------
setheron
I think a single sensible makefile should be provided.

------
dingo_bat
I tried to get into rustlang and golang but both of the languages are so
_opinionated_ about how I should structure my workspace, how I should manage
dependencies that I gave up. I'm used to cmake and C/C++ where I can do
whatever I lik, however I like. I have to give go a try again after reading
TFA.

~~~
goobynight
They're opinionated because they're designed to be used by large teams
building large software for a corporation.

In these environments, you usually end up with an opinion anyway, so they
built it right in, so people could skip the debate stage.

