
Go Modules in 2019 - baijum
https://blog.golang.org/modules2019
======
treve
One of the thing that's always been a bit off-putting about the Go community
and leadership is that it seemed that when I came across things that I
perceived as flaws in the tooling or language, I often felt told off and was
made to feel that it's me who is wrong for a variety of reasons. Examples of
this include GOPATH, the package infrastructure, error handling, lack of
generics. Rob Pike disagrees so I must be wrong.

It's kind of satisfying to see that the "Go way" wasn't the best way after all
and that some of these things are actively being addressed. I hope it makes
the community a bit more welcoming as well.

~~~
jonahx
Contemptuous scorn seems to be the officially-sanctioned strategy for
responding to reasoned critique or even questions about "the go way" within
the community.

My favorite example here is this arrogant dismissal of the idea that a one
line _expression_ , rather than a 5 line mutating statement, might be
preferred for conditionally setting a value:

    
    
        if expr {
            n = trueVal
        } else {
            n = falseVal
        }
    

> The if-else form, although longer, is unquestionably clearer. A language
> needs only one conditional control flow construct. [0]

...

    
    
       n = expr ? trueVal : falseVal
    

"Unquestionably"

[0]
[https://golang.org/doc/faq#Does_Go_have_a_ternary_form](https://golang.org/doc/faq#Does_Go_have_a_ternary_form)

~~~
kardianos
The authors of Go have stated they themselves sometimes miss the ternary form,
but have seen it abused too much (deeply nested ternary).

This isn't an insult to programmers, it is an option that clarity of code is
more important then continence. Thus all if and for statements also require {}
brackets.

~~~
jonahx
This would be a reasonable position:

"While the ternary is often clearer, we chose to sacrifice expressiveness and
brevity for the sake of preventing abuses, which we found were all too
common."

But that is not the claim being made in the FAQ.

> Thus all if and for statements also require {} brackets.

Indeed this rule seems to spring from the same philosophy. It is most
certainly _not_ a preference for clarity, though. It is a preference for
_consistency_.

The philosophy is: "We're giving up expressiveness and brevity because in our
experience most people can't be trusted to not shoot themselves in the foot."

This choice would be much more palatable if they were honest about it. But
instead, they take the road of insisting that the verbose consistency is
actually _clearer_ , which it isn't, at least in many people's opinions.

~~~
Merovius
> It is most certainly not a preference for clarity, though. It is a
> preference for consistency.

FTR, with "most certainly" you're committing the same fault your accusing the
Go team here. You might not think it matters for clarity. I, at least,
disagree.

> in our experience most people can't be trusted to not shoot themselves in
> the foot.

I don't understand the difference between this suggested phrasing and talking
about clarity. I'd argue you tend to shoot yourself in the foot if and only if
you aren't clear about what you're doing. Clarity and lack of footguns seem
directly correlated.

~~~
jonahx
> FTR, with "most certainly" you're committing the same fault your accusing
> the Go team here.

Fair enough. It's too late to edit.

> I don't understand the difference between this suggested phrasing and
> talking about clarity. I'd argue you tend to shoot yourself in the foot if
> and only if you aren't clear about what you're doing.

This is partially true but it's not that simple imo.

In my OP, the ternary version is clearer. But _if_ you allow it, then you also
open the door to nested ternaries, and foot shooting.

Allowing single line if statements (as, eg, ruby does) also allows for clearer
code, at the expense of consistency.

~~~
Merovius
> Allowing single line if statements (as, eg, ruby does) also allows for
> clearer code, at the expense of consistency.

This (or rather allowing conditionals without braces) was what I was referring
to. They are a foot-gun, because of the lack of clarity. It's a common source
of bugs in C/C++ code that a developer thought they'd add a line of code to a
conditional block or loop but didn't, because of a lack of braces. Requiring
braces unconditionally make it always unambiguous and obvious what block a
given statement belongs to.

Obviously YMMV - this is, as most discussions in programming, a matter of
opinion. Which was my point :)

~~~
jonahx
I'm aware of the reasoning. As you say, in C people would carelessly edit:

    
    
        if (condition)
          doSomething
    

to

    
    
        if (condition)
          doSomething
          doSomething2
    

So fair enough, you protect against that. You cannot therefore conclude that:

    
    
        if (condition) {
          doSomething
        }
    

is clearer or that 1 line if statements aren't clear. First of all, you've
purchased your insurance at the cost of brevity. Which is quite a high price,
especially when you have to use them _constantly_ to write:

    
    
        if err != nil {
          return err
        }
    

and when it forces all of your 1 line conditional expressions to be 4 - 6
lines, per my OP.

And there are solutions that allow you to buy your insurance without such a
high price. You can invert the position of the if, as ruby does. You could
make a rule that when you don't have braces you have to write your statement
on the same line. The point is this isn't the _only_ way out. And clarity is
not the same as preventing one specific kind of error, which occurs only in a
context which could be changed as well.

~~~
rufio
on that note: here's an example of a very costly example of this exact
mistake, Apple's famous "GoTo Fail" for anyone that isn't aware of it:
[https://dwheeler.com/essays/apple-goto-
fail.html](https://dwheeler.com/essays/apple-goto-fail.html)

------
plg
I really want to get into Go, I like that it's opinionated, I like that it's
compiled, I like that it's garbage-collected, I like that coroutines are
built-in.

My primary use case is scientific computing, both data processing and
interactive visualization.

I know Julia is an option as well.

For reasons I don't want to bother getting into I dislike Python.

Thoughts?

~~~
sbinet
others have already mentioned Gonum:

\- [https://gonum.org](https://gonum.org)

Gonum is _almost_ on par with _e.g._ NumPy/SciPy (most notably lacking: ODEs).
So still ways to Go but it's getting there.

Go-HEP is my attempt to bring a few High Energy Physics oriented packages to
particle physicists:

\- [https://go-hep.org](https://go-hep.org)

I've also written a few words on why I think Go is great for science:

\- [https://sbinet.github.io/posts/2018-07-31-go-hep-
manifesto/](https://sbinet.github.io/posts/2018-07-31-go-hep-manifesto/)

TL;DR: Go is great b/c it brings great s/w engineering practices and a s/w
engineering-friendly environment to scientists.

Admittedly, generics will change how packages are written. So _some_ code
churn will take place when/if they land, but the Go community learned the
lessons from Python2/3 and Perl5/6\. Expect a better migration path.

Lastly, I guess the 2 remaining weak points of Go are:

\- runtime performances sub-par wrt C++ or Rust

\- GUIs (which may or may not fall into "interactive visualization")

That said, the Go community worked on a Go kernel for Jupyter:

\-
[https://github.com/gopherdata/gophernotes](https://github.com/gopherdata/gophernotes)

\- [http://gopherdata.io/](http://gopherdata.io/)

hth, -s

~~~
tptacek
Gonum is neat, but to the previously-made point about Go's type system making
stuff more painful than it needs to be in this application: gonum's linear
algebra is defined over float64 and int, which is problematic if you need
arbitrary precision.

~~~
sbinet
I guess it's scientific domain dependent. I've very rarely needed something
else than float/double in my C++ days or REAL*8 in my F77 ones.

sure, when you need it, you need it. but float64 caters for a good 99% of my
usual work day.

From a user POV, seamless installation of packages is a great boon. From a
grid/cloud operator POV, static binaries are great too.

~~~
tptacek
These are things that are great for product development and devops and not in
fact all that valuable in scientific computing, which is a reason why so much
of it gets done in Python.

~~~
sbinet
> These are things that are great for product development and devops and not
> in fact all that valuable in scientific computing

I disagree. Again, this may very well be science-domain dependent, but in High
Energy Physics (where, finally, Python is recognized as a mainstream language,
BTW) many -- if not all -- of the pain points that slow down undergrads, PhDs,
post-docs and researchers at large, are these Go features.

yes, the time from "idea" to "little script that shows a bunch of plots" on a
subset of the overall data is definitely shorter in Python (exploratory
analysis is really really great in Python). but, at least for LHC analyses,
python doesn't cut it when it comes to Tb of data to swift through,
distribution over the grid/cloud, ... you die by a thousand cuts. and that's
when you are alone on one little script. LHC analyses can see well over 10-20
people assembling a bunch of modules and scripts. You really long for a more
robust to (automated) refactoring language than python, pretty quickly.

------
jrockway
Does anyone know what the story is for binaries generated by things your
project depends on? npm versions them and lets you invoke them from the
project with "npx"; I wish something like this existed for go. (protoc-gen-go
is the main thing I want; if your global version gets out of sync with the
version in go.modules, the generated protobuf doesn't compile.)

I am glad to see that the go team is working on cleaning up the tooling
situation. I used gocode, then started using modules, found that the primary
version was unmaintained, and had to switch to a different fork. I believe I
also needed a different version of goimports for a while. Having all this
tooling unified into the langserver maintained alongside go sounds wonderful.
I hope other languages do the same thing.

~~~
Merovius
This is exactly why I personally think efforts for language-specific package
management in general are misguided. This is a solved problem in general-
purpose package-managers (apt, dnf, pacman,…), for all their flaws. I don't
understand why we need to re-invent that over and over again, including all
the duplication of effort to re-package everything over and over again…

~~~
pjmlp
Because a language-specific package management work in all OSes supported by
the language, while OS specific packages work in a single distribution, or not
all in OSes that don't offer such support.

So instead of creating M packages * N OSes, we do it just once.

~~~
Merovius
> So instead of creating M packages * N OSes, we do it just once.

No, creating M packages x N OSes is _exactly_ what we do. In fact, we create
M+N package- _managers_ x N OSes - and _then_ also create MxN packages.

If you can write a language-specific package manager for all OSes, you can
also write a non-language-specific package manager for all OSes, so I the "it
doesn't work in all distributions" argument is just a symptom of my complaint.
Instead of working towards a cooperative packaging solution where the effort
of packaging can be re-used, we continue to create more and more special
snowflakes and fragmentation.

~~~
pjmlp
Good luck creating a package format that works across iOS, Android, Red-Hat,
SuSE, Debian, Ubuntu, ..., IBM i, IBM z, Aix, HP-UX, Solaris, Windows, Zephyr,
Yoctos, RTOS, Integrity, mbed, MicroEJ, BSD variants, Unisys ClearPath,
VxWorks, QNX, macOS, Tizen, Jolla, ChromeOS, Fuchsia and several others that I
am unaware of or was too lazy to keep adding entries for.

My C++, Java and .NET packages work everywhere there is a toolchain available.

~~~
Merovius
> Good luck creating a package format that works across iOS, Android, Red-Hat,
> SuSE, Debian, Ubuntu, ..., IBM i, IBM z, Aix, HP-UX, Solaris, Windows,
> Zephyr, Yoctos, RTOS, Integrity, mbed, MicroEJ, BSD variants, Unisys
> ClearPath, VxWorks, QNX, macOS, Tizen, Jolla, ChromeOS, Fuchsia and several
> others that I am unaware of or was too lazy to keep adding entries for.

Can you explain why that would be a problem? It's certainly not a technical
one, none of these are special when it comes to versioning or dependency
management of software. I can see that there's a social/political problem -
which is exactly what I'm talking about.

~~~
pjmlp
It surely is a technical one above any political willingness.

A package format that supports all OS system paths, installation processes,
difference between build time/dev time/deployment time, language compilation
toolchains, compiler flags, ways to address hardware resources,OS specific
deployment processes, ... is bound to the lowest common denominator for any
chance of success.

Thus forcing everyone that needs something beyond that lowest common
denominator to implement their own workarounds, thus we are again back to
language package managers.

~~~
Merovius
> is bound to the lowest common denominator for any chance of success.

ISTM the "lowest common denominator" is a superset of everything current
language-specific package-managers support and a subset of anything a current
language-agnostic package-manager supports (pretty much definitionally). So
ISTM that this is a net win - easier to build than APT/DNF/Pacman… and yet
more useful than npm/stack/pip/…

In particular, I don't know any currently existing language-specific package-
manager that supports what I'd call the GCD of package-management (e.g. none
that I can think of supports actual OS-specific installation of packages) so
that clearly isn't even a requirement.

> Thus forcing everyone that needs something beyond that lowest common
> denominator to implement their own workarounds, thus we are again back to
> language package managers.

FWIW, a) part of the political and social problem is to talk more honestly
about what "needing" really means and b) no, that's not at all "back to
language package managers". You can have a layered design, e.g. splitting the
"building" and the "installation" part, thus letting languages implement their
workarounds in their dedicated building layer and letting OSes implement their
workarounds in their installation layers. You just need to actually sit down
and talk about the interface needed between the two (and the sets of layers
actually needed, which will be >2). Which no one seems really willing to do.

------
candiodari
Maybe it's just me but I hate this option. Are we really, really trying to
optimize the downloading of source code ? All 5 megabytes of it ? Why would
you do that ?

I regularly find myself in a position that this system would make impossible:
I need a few custom changes in public available libraries. An extra method. A
bugfix. What have you. This makes that impossible using the default method.

What we want is consistent builds. That means a build that happens, with the
same source, with the same ... every time, always, on everybody's workstation.

~~~
pokstad
Go modules still allow for changes to local copies. You are not required to
use a proxy to use modules. Also, you can always fork a public project and use
a replace statement in a module to redirect an import path to your customized
module. Check out this related tool:
[https://github.com/rogpeppe/gohack](https://github.com/rogpeppe/gohack)

~~~
candiodari
And _another_ configuration file that you (and all tools) need to know about
pointing at _yet another_ global directory (global for all your projects).
That override file, of course, _does_ have to be in your project folder and is
yet another thing to take into account.

If I want to share things, I am aware of links in filesystems, thank you very
much Go authors. "go.mod" overrides ... I wish golang did _not_ try to solve
yet another non-problem _badly_.

------
pjmlp
Nice overview, however a small correction to the statement about Maven
repositories.

Decentralized support has been a thing since the last decade.

~~~
emersion
What do you mean by "decentralized"? Do you mean someone can add a third-party
repository and download packages from there? (Then it's been the case for
other environments too)

I think decentralized is meant as "there's no central repository" here.

~~~
pjmlp
Same applies to Maven, in fact many enterprises forbid access to Maven
central, while curating an internal one.

~~~
Merovius
> Same applies to Maven, in fact many enterprises forbid access to Maven
> central

"Maven central" implies there _is_ a central repository though.

~~~
TheDong
There are two different meanings of 'central' in play here, 'center' as in a
town square, and 'center' as in the centralized component of a system.

It does not imply that there is a central repository when using 'central' to
mean 'a unique, difficult to replicate, or otherwise somehow special component
in a system' as in 'centralization'.

Central in the case of 'Maven Central' means something closer to 'default' or
'town square' or such.

Saying that 'maven central' implies maven isn't decentralized would be the
same as saying "Most go projects are on github, so go can't be hosted in a
decentralized way because github is 'golang central'".

~~~
Merovius
> Saying that 'maven central' implies maven isn't decentralized would be the
> same as saying "Most go projects are on github, so go can't be hosted in a
> decentralized way because github is 'golang central'".

It's not, though. Because github _isn 't_ the default. You have to explicitly
specify that you want a project from github every time you use it.

The difference might be minor from a technological standpoint, but it means a
lot in terms of how centralized things become culturally.

~~~
pjmlp
Just like enterprises have their official .m2/settings.xml file pointing to
the actual default Maven repository.

------
Boulth
> One of the most important parts of the original design for go get was that
> it was decentralized

That's one of the things I like in go most. Having decentralized packages via
domains and URLs and then index them (godoc.org works very well), it mirrors
the design of the Web. In contrast npm, Rust and others that are tightly
coupled to one site feels like Google AMP - centralizing and hosting
everything in one place.

~~~
almostdeadguy
Cargo (in the nightly channel, but eventually will become stable) and npm can
both use alternative package indexes and vendored dependencies, so they are
not tightly coupled. Having a standardized package index as a database is what
allows efficient dependency solving, which Go hasn't considered providing up
until recently.

> that anyone should be able to publish their code on any server, in contrast
> to central registries such as Perl’s CPAN, Java’s Maven, or Node’s NPM.
> Placing domain names at the start of the go get import space reused an
> existing decentralized system and avoided needing to solve anew the problems
> of deciding who can use which names. It also allowed companies to import
> code on private servers alongside code from public servers.

lol statements like this are infuriating. Go is no more decentralized than
those languages, unless you consider git hosting something people in standard
practice do on their own (survey some of the top Go dependencies and see if
that holds true). They simply lack a package index, but the cited language
package managers both have those and allow you to host your own.

------
emersion
It seems like there's a mistake in the diagram. The "notary → mirror" arrow
should be replaced with a "notary → go command" one, because the go command
shouldn't trust the mirror when it comes to cryptographic hashes.

~~~
pokstad
I think the mirror can pass through the public signature provided by the
notary. That cannot be spoofed if you have a trust chain for the notary to
ensure the mirror has not tampered with the module.

------
dana321
A centralized module index will be nice, i tend to end up searching github.com
which leaves out all the other sites or locations that could have a module
that solves my problem.

~~~
xyzzy_plugh
I very much do not want this. godoc.org has decent search and I've never had
any complaints.

~~~
Merovius
Note that there's a difference between "central registry" and "central index".
In particular, I don't see any downsides from what they are changing -
godoc.org is already, for all intents and purposes, a central index, so I
don't see how anything changes in that regard.

------
teacpde
So the mirror will serve both the package and signed hashes, what if the
package has been indexed in the central index but not yet signed by the
notary?

~~~
emersion
Maybe asking a hash to the notary will actually make the notary fetch and hash
the package if it doesn't know about it yet?

(Related:
[https://news.ycombinator.com/item?id=18717766](https://news.ycombinator.com/item?id=18717766))

~~~
teacpde
That works, but slows down the mirror? I guess it is okay since it should only
happen once.

------
spullara
It is amazing that it took them this long to realize they were entirely wrong
about how dependencies should be distributed. Since it was obvious from day 1
to most people from other ecosystems maybe they shouldn't have so much NIH.
We'll likely get exceptions and generics soon as well. What a waste of time.

~~~
chimeracoder
> It is amazing that it took them this long to realize they were entirely
> wrong about how dependencies should be distributed.

It didn't take "this long"; the Go team openly admitted the need for a better
approach at the very first GopherCon back in 2014, nearly five years ago.
(That was not the first time, either).

It takes time to do all that work.

~~~
spullara
If you realize something is wrong a couple years in and it takes 5+ years to
fix it, then that seems like a long time to me.

~~~
geodel
In between community developed many solutions when official solution was
lacking. So it's not like people were twiddling their thumbs for 5 years. Also
check how long Java took to get official module solution. Work on that started
in 2008 and solution was delivered in 2017.

~~~
spullara
Hah. They don't have one now. The de facto solution is Maven. Java itself
doesn't actually have a module system like the one discussed for Go. It is
merely a access control system rather than a dependency system.

To be clear, I give the Java people shit about it all the time and was on the
original Java module system JSR that was supposed to deliver it. In the end,
the Sun people yanked the dependency part and shipped a module system that is
really only useful to the JDK.

