
Go Packaging Proposal Process - zalmoxes
https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8
======
aikah
Go ecosystem is a complete mess. I understand what Go maintainers tried to do,
but Go isn't like Gnome lib,or Linux core lib where dependencies are stable.
And the "just vendor" stuff pushed by some Go pundits is stupid. Clutter or
Gtk don't vendor Gdk or Glib. So Go right now is about everybody reinventing
the wheel, because you can't write reproducible build with 3rd party packages
thus create an ecosystem with Go get. The solution ? deprecate Go get in favor
of a new tool that supports at least fetching tags for packages. Project based
build Tools (like Gb) work for apps, they don't work for libraries. Glide and
co have a big problem : everybody is using a different package manager that is
incompatible with others.

The question isn't being solved yet because to solve it seriously it would
require to change how the language works at first place.

~~~
maerF0x0
> And the "just vendor" stuff pushed by some Go pundits is stupid.

I see it as an excellent way to allow me to ensure that no matter what my
project continues to build and work. And dont say "just pin a version" because
there have been packages that lose or update their tags and break things, or
the distribution channel (npmjs, github, w/e) go down and suddenly I am up a
creek for a deploy/build.

When I vendor, I have what I need to build and deploy my code with a lot fewer
points of failure.

~~~
nyan4
...until the unfortunate end user that needs to run tenths of systems runs in
a security issue. Then the admins cannot possibly learn how to fork, patch,
rebuild, test, deploy in 20 different languages. And they cannot rely on
security updates from Linux Distributions because they installed vendorized
code blobs.

~~~
zwily
This has more to do with the lack of dynamic linking than vendoring.

~~~
nyan4
The two are very connected. Yet, even with dynamic linking, the libraries
could be always and _only_ bundled with the application, and the application
could be designed to work against the bundled versions and not even tested
against other versions. On a practical level this makes it extremely expensive
to perform updates.

------
mfer
Hi folks, My name is on that list and I work on one of the Go package managers
(Glide - it's like npm, crates, composer, bundler, etc... but for Go...
[https://glide.sh](https://glide.sh)).

If you have questions or input those who are working on this problem are
listening.

~~~
scrollaway
[This question is purely informational and doesn't directly relate to golang,
but I want to ask someone who has been involved in domain-specific package
management.]

Why is the protocol side of package management not a solved problem already?

At least Go reuses the git protocol to its advantage. But the general problem
of package management applies to a lot of things.

"There is a package with a name and a description. That package has versions,
which can be greater/lesser than others. Those versions may have deltas from
one-another. They contain one or more files. Those files must have their
integrity checked. etc, etc..." -> Why does every single language reimplement
that?

Nevermind languages actually. Why does _everyone_ reimplement that?
Firefox/Chrome extensions, Android/ios app stores, a bazillion different game
addons and app-specific stores...

We have a mostly-identical problem being solved a million different ways.
pypi, npmjs, glide, they should be running the same server software. It'd be a
first step towards standardization and avoid a bunch of mistakes repeated
every time a new solution comes out.

Is anyone even trying to do something about this? (and before anyone says "do
it yourself", a package management protocol _is_ something I used to work on,
not anymore. It keeps getting worse - it's not just a technical problem, also
a political one.)

~~~
shurcooL
> That package has versions

This is a statement that doesn't always apply.

For example, some packages don't have versions, they only have a single
version: the latest version. It's a moving target, but it's there. Older
versions are just an unimportant and unsupported implementation side-effect.

Of course, many packages do have versions, but this one factor makes a big
difference on how you look at vendoring.

~~~
vec
First, your described software still has versions. Each daily snapshot or git
commit or whatever is a new version. There's no reason you can't put an auto-
incrementing version number on each release and call that your version number.
You can even stick a "0." on the front of it and be SemVer complant.

Second, just because some people do this doesn't mean it's a good idea. If
you're writing a full application and just want to distribute it, do whatever
you want with your versioning (although using a dependency management tool to
distribute your webapp or CLI tool is probably not a great idea in the first
place), but if you are writing something to be included in another project
please use a sane versioning system. Anything else is selfish and
disrespectful of your users.

~~~
bassislife
In Go, each version corresponds to a given import/Pkg path.

If you want to change version (major), you need to create a new package.

The advantage is that you do not have to download a manifest such as
package.json or whatever. It also provides a constraint on library authors to
provide a backward compatible API for their published package.

Eventually, I guess we could use go/types to enforce/check the API backward
compatibility requirement.

The main issues for now are:

* make sure that packages importing a same vendored one are able to use the __same __latest minor version in use.

* allow a project based approach to go get for people who need reproducible builds (or alternatively allow for multiple $GOPATH and make being able to switch between them easy)

------
twblalock
I never understood why Go conflates _where_ something is (its Github url) with
_what_ something is (a specific version of a specific bit of code, what Maven
calls an "artifact," and what lots of other solutions call a "package," or
"module," or "gem", which has a version and never changes).

This creates a lot of problems that simply don't need to be there:

1\. Versioning dependencies is pretty much impossible unless you make your own
copy of them, and keep them updated. Good dependency management solutions do
not require you to "vendor" your dependencies. They enable reproducible builds
by keeping every version of an artifact around.

2\. Github could change its URL schema, and most Go projects would stop
compiling.

3\. The maintainer could delete the Github repo.

4\. Source control, specifically Github, is tightly coupled with dependency
management, and that should not be necessary.

These are not unsolved problems. Maven solved every single one of these
problems years ago. I don't expect people to use Maven to build Go
applications, but clearly there is an opportunity to use Maven's good ideas
and build a better dependency management system for Go.

I have to say, in defense of Maven, that it is very robust: artifacts are
mirrored in many places, including in your own local repo; versioning works
well, including availability of older versions; and nobody can screw you by
deleting package or changing its Github url. My company uses Maven for Node.js
as well as Java, because we've had so many problems with NPM.

I should also point out that the expectation that the core Go team should
develop a dependency management system is a bit odd. The dependency management
systems for Java, Ruby, Python, Node.js, and Perl were all developed by the
community, not the core team. Some of them were later absorbed by the core
team, but the point still stands.

~~~
slrz
You, too, seem to misunderstand the concept of a Go import path. It's a string
that identifies a package. It's not a URL. To provide for sane namespacing,
people tend to use DNS names and URL paths to name their packages, just like
they do with Java package names. There's no requirement that there's any
correlation between the package import path and where you fetch that package's
code from.

Thus, your statement: _2\. Github could change its URL schema, and most Go
projects would stop compiling._

is wrong. No single Go project that compiled before would now fail to do so.
The common build tools use the import path to determine a path on the _local
filesystem_ where source or object code may be found. Neither a change of
Github's URL schema, nor a repo deletion by some maintainer would cause your
build to fail.

The thing that would break is of course the re-fetching of source code from
that repo, but that's kinda obvious, don't you think? You can't clone from a
deleted Git repo. A common solution to that problem (language-independent) is
to mirror the code somewhere else.

Now, there's one single thing in the standard Go tooling universe that
interacts with remote Git repos. It's 'go get', a small convenience wrapper
around several VCS commands. It allows you to clone a remote repo to the right
place on your disk while using the same syntax for Git, Mercurial and SVN
alike. That's basically all it does. Yet, some people expect it to be some
kind of elaborate package management system. It's not.

------
the_duke
Looong overdue...

With Go being a new language (1.0 in 2012), there is no excuse for not having
had a solid pkg solution in 1.0.

It's one of the things holding Go back, in my opinion, producing makeshift
solutions like Glide and gopkg.in, which are only minimally adopted.

I hope they also specify some mandatory versioning scheme (like npm with
semver), because there are so many Go packages out there which don't even do
any versioning!

~~~
slrz
The idea that somehow every single language and community needs its own crappy
package management systems is just utterly broken.

There's no reason you can't use the same package format and tools for code
written in multiple languages. You might get some additional convenience by
having small language-specific plugins (like the automatic dependency
extractors in RPM) but that's just a tiny amount of the necessary tooling. The
vast majority is not language-specific at all.

With these language-community systems, there's also the problem of depending
on code written in another language. My Go program might well depend on a C
library, just as my C program can require a shell script.

~~~
rat87
Unfortunately the idea that a new language meant for widespread use can ship
without its own package management systems is even more broken.

Generally languages need a good tool to allow (nested) dependencies that has a
simple (preferably declarative) dependency file.

Would it be nice to be able to have a cross language platform tool(jvm tools
do generally play nice across languages in my understanding) like nix that
also allows setting up other non language specific dependencies? Yes. But I'm
not sure any are good enough/simple enough yet. Also language ecosystems tend
to avoid non language dependencies (unless they're so common theyre typically
installed with the language like lxml/libxml2) or have crappy ways to try to
build c dependencies. So this is if not solved partially worked around.

These package managers are mostly for development, not deployment although
frequently they are also used to build or even deploy code.

It would be nice for each language package manager to have plugins to generate
self contained deb/rpm/ect. for deployment or even fancier to allow some
dependencies to be (assuming it's not a language that prefers static
linking/bundling) satisfied by system packages. But these take time and effort
to write.

------
niftich
I'm new to Go and I read some documents [1][2][3] on what 'vendoring' means
but I still don't understand.

What problem does it solve? Why is it needed?

Is there an 'explain for a 5-year-old' version, or better yet, an 'explain for
someone who is used to other packaging systems that may not be entirely
elegant but work pretty okay most of the time, like Java's Maven or Node's
npm'

[1] [https://engineeredweb.com/blog/2015/go-1.5-vendor-
handling/](https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/) [2]
[https://github.com/golang/go/wiki/PackageManagementTools](https://github.com/golang/go/wiki/PackageManagementTools)
[3]
[https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia...](https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia6NkmB024MUa4T3SJeh8)

~~~
twotwotwo
It's common practice for folks building Go binaries (but not libraries) to
check in the code for their dependencies. So whenever you build Docker, you
get the same thing unless Docker made a change (either by changing Docker
itself or explicitly checking in an upgrade to a dep).

There are a few nice things about it; vendoring means you won't have a
dependency on other repos or archives being available, and if you're a binary
maintainer you can see source changes in your own code and that of your
dependencies side-by-side if you're, say, investigating a bug that came up
between certain revisions.

The not-so-nice things are pretty much what you'd think. There isn't a command
to say "give me upgrades, but not ones that the author flagged as breaking
changes". If you're writing a _library_ that is depended on by other
_libraries_ there is no one conventional way to safely do breaking changes, so
there's a lot of effort to avoid those changes plus sometimes hacks like
different repo paths for different versions.

Part of the history is that, internally, Google's got one version of third-
party code in the latest rev of their source control tree, by company policy.
You upgrade something for one, you upgrade it for the whole codebase. Sounds
daunting, but they have lots of testing, canary servers, and so on to catch
problems. There is a great talk about some of this by Rachel Potvin (can't
recall if it covers the third_party/ stuff specifically):

[https://www.youtube.com/watch?v=W71BTkUbdqE](https://www.youtube.com/watch?v=W71BTkUbdqE)

So Go's initial model didn't consider multiple versions of packages because
Google had approached upgrades a different way internally. Of course, unlike
Google, Go users on the outside will never have any central mechanism to keep
all the world in sync when there are API changes. Hence the conversation today
about package management.

~~~
twblalock
I think another reason they didn't consider proper versioning is that only a
few of the Go core developers have a background in languages with a good
dependency management ecosystem. C and C++ have never really had that, and
Python never really has either, despite what fans of it seem to think.

I think the bottom line is that the Go team simply doesn't know how good the
ecosystem is in a lot of newer languages. I see this reflected in other
decisions in Go as well, such as the absurd amount of boilerplate needed to
manage some of the Goroutines, and the way string are handled.

~~~
d0vs
> I see this reflected in other decisions in Go as well, such as the absurd
> amount of boilerplate needed to manage some of the Goroutines, and the way
> string are handled.

Can you expand on this?

------
howeyc
I'm exited to see this progressing.

I'm grateful for anyone who serves on the committee, as it could be a
thankless job (some people may not like the agreed upon tool/process, but
hopefully a small group that doesn't feel entirely disenfranchised).

I really hope whatever they come up with is welcomed by the community and
projects migrate to using it. Basically becoming the "Standard Go Way" to do
things.

------
kaeshiwaza
I hope you will find a "Go solution". I mean stupid simple (for most of the
time).

~~~
nameless912
Actually though, I'm hoping for exactly this.

One of the things I _love_ about Golang is that it's opinionated; gofmt forces
your code to conform to a uniform style, the linters are top-notch, and the go
command line tool builds your code in exactly one way. Some of the decisions
made (the lack of generics sticks out) are frustrating at times, but the
language authors having opinions stops many developer quibbles from becoming a
problem. Now it's time to add packaging to this list.

When we use go at work, everyone conforms to the same coding standards,
everyone uses the same documentation tools, everyone deals with the same build
system, and everyone (for the most part) writes cleaner, more uniform code
because of it. I get why some developers hate Golang: it forces lots of its
own opinions on how to do things on you, but in exchange it removes almost
completely the bikeshed problem; everyone _has_ to paint their bike shed the
same color, so we can move onto the meat of solving the problems. I personally
have been fine with go get and the vendoring tools people have created because
we're using almost entirely internal libraries so getting the latest version
of the master branch is usually the right choice, but I understand the need
for a better package management solution. I just hope, like the parent
commenter, that the go package solution is as opinionated as the rest of the
language; give us an easy, sensible default so that the tooling gets the hell
out of my way. I _hate_ working on C++ and Java projects at my job because of
how heavy the tooling often is (CMake and Maven, respectively), and that's
just for the build system; forget autocomplete or linting or refactoring
support unless you're in a commercial IDE. On the other hand, Go is simple
enough that emacs is powerful enough to take care of every IDE feature I could
ever want. Hell, the command line and nano is plenty to edit Go code, because
the build system works exactly one way, and that is so incredibly important to
me.

So yeah, I want to see a _good_ solution. But I will take a less complete
solution over one that adds more complexity.

------
zellyn
btw, I also wrote up a brief "State of Go Packaging" doc you might be
interested in reading:
[https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy...](https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk)

------
mmkx
This is needed badly. I am a huge Golang advocate, but I refuse to use the
"github.com" package naming/folder setup mess. It's just not elegant.

~~~
jvehent
Projects I care about use a domain I control. It's not that hard to setup:
[https://jve.linuxwall.info/blog/index.php?post/2015/08/26/Ho...](https://jve.linuxwall.info/blog/index.php?post/2015/08/26/Hosting_Go_code_on_Github_with_custom_import_path)

We even do it with just an S3 bucket for go.mozilla.org:
[https://go.mozilla.org/](https://go.mozilla.org/)

~~~
mfer
I'm a fan of that as well. Had not considered using S3 for that. Do you know
of a good writeup for handling "?go-get=1" paths differently from html paths
in S3? I should likely know this but a quick pointer would be helpful.

~~~
jvehent
I have not looked into this.

The S3 method is kinda experimental. It works for first-level packages but
fails for nested packages. We need to look into setting up redirects for
those.

The best way to do this yourself is to run an instance of
[https://github.com/niemeyer/gopkg](https://github.com/niemeyer/gopkg), but
that's not #serverless anymore ;)

------
peterwwillis
If you want to model a real package manager, don't do a comparison of existing
tools, all of which have flaws. Start with the feature set of RPM, and then
ask a professional software packager + developer what is missing.

In order to have a feature-complete package manager (that is, it does
everything you will concievably need) you need at least the following:

    
    
      - Unique file/object tracking
      - Upgrade & Rollback
      - Dynamic string interpolation (file paths, for example)
      - Identification of architecture and OS
      - ACL and extended permission support
      - Tracking of both source and binary dependencies & versions
      - Before & after script hooks for every single operation
      - Cryptographic signing & verification
      - Support for multiple central & distributed repositories
    

This is, of course, completely different from a build system, but has a few
features in common. You may not need a package manager at all; you may just
need a feature-complete build system, which is how you end up with CPAN.
(There is no Perl package manager, though PPM attempts one, and Carton is just
a bundler)

------
randomsofr
I hate using the url for the namespace.

~~~
themihai
Do you prefer :: instead of / ? Btw go doesn't use URLs, just domain and path
(e.g randomsofr.io/pkg/subpkg )

------
0xdeadbeefbabe
So what's wrong with vendoring all the deps and version controlling
everything?

Edit: Seriously, I'm not being a jerk.

~~~
mfer
vendoring is dependency storage. Management, especially when you have
transitive dependencies (deps of deps), is more than storage. We need to solve
the non-storage related problems.

Are you familiar with the diamond dependency problem? That's one example.

~~~
0xdeadbeefbabe
I'm not familiar with the name, so I looked it up thank you.

Program A depends on B and C which both depend on the stdlib (D). That seems
pretty common. I actually have this going on now, but so far so good.

I suppose D' (the new version of D) might break B while improving C and then
I'd have a problem, right?

------
leonardinius
I wonder why I was able to edit document on first open. :o Seems to be solved
quite quickly.

~~~
zellyn
Heh. Thanks for the warning. I just changed my document (mentioned at the
bottom) to comment-only too.

