I started poking around and noticed that Go 1.13 now defaults to the Golang Proxy to fetch modules. This means a proxy, governed by the Google Privacy Policy, is now capturing everyone's module usage by default. Unless you change settings this includes proprietary/corp stuff.
"go env -w GOPRIVATE=*.corp.example.com" was added to make it as easy as possible to configure private modules. If the environment is not set up, just the name of the module will reach the Google services, it will not be published, and an error will be returned.
Anything else would have make it possible to bypass the protections of the new Checksum Database (https://golang.org/design/25530-sumdb), a state-of-the-art authentication system that provides auditable proof that the operator is behaving honestly.
For an example of why the Mirror is valuable, beyond the 10x performance gains (depending on number of dependencies and connection latency) and security gains from not using git on the client, look no further than git.apache.org, which has been down for days breaking a number of builds (https://status.apache.org/incidents/63030p4241xj).
I really think this architecture brings together the best of the decentralized module ecosystem that Go had (with its lack of regularly compromised registry accounts, and no mismatch between code and package origin) and the centralized registries of other languages (with their better performance and reliability).
Particularly proud of how the Checksum Database provides even more security than the centralized registries (because it's cryptographically accountable for its actions, while registries can sign anything they want) without requiring authors to manage keys, register an account, or change their workflow at all.
[Note that I work on the Go team, and co-designed the Checksum Database. But of course I don't speak for the company.]
EDIT to add: I should also mention that the sumdb allows the use of any untrusted proxy, easily configured with "go env -w GOPROXY", for users that can't or don't want to use the Google provided one. The proxy can also opt-in to mirroring the Checksum Database, in which case no user requests at all need to reach the Google services, while still providing all the integrity guarantees that proxy.golang.org users benefit from.
First programming language that I've had to read and understand a privacy policy to use.. and consider that it may change in the future.
The idea that the language I'm programming in now reports anything back to google is distressing.. and I say that as someone who has been programming in Go for about 7 years. What right does google have to collect usage information from modules hosted on github (or elsewhere)?
I checkout the code for the modules, review it, and commit it into my own repository. I don't need you to speed up the rare git checkout of a module. Not that I think some minor speed improvement is worth my privacy.
You talk about decentralization... while centralizing the entire thing around Google. You talk about git.apache.org being down, while you're willing to bring down the entire ecosystem across the board with your own special server.
This "feature" should have an option to disable it.. at the very least.
This is the first language in my 23 years programming that reports my usage back to the language authors. It's unprecedented. And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow.
> This "feature" should have an option to disable it.. at the very least.
The Go 1.13 Release Notes lead off with documentation links for how to disable and otherwise configure this:
"See https://proxy.golang.org/privacy for privacy information about these services and the go command documentation[1] for configuration details including how to disable the use of these servers or use different ones. If you depend on non-public modules, see the documentation for configuring your environment[2]."
> This "feature" should have an option to disable it.. at the very least.
You should be able to. It's not the ideal solution (or recommended for most use cases), but you can by setting GOPROXY=off. GOPROXY=direct will force it into its previous behavior.
The same should be true for the checksum DB with GOSUMDB=off. I don't remember specifically and can't find the page its options were documented on.
IMO, using an environment variable for that is not robust. If the environment variables have been cleared (for instance, due to "su -" or similar), or if they have never been set (for instance, because you just checked out the code on a new machine and forgot to configure the environment variables there, or because you did remember to configure the environment variables but forgot that startup scripts do not take effect on an already running shell), the defaults will apply. Using a configuration file within the source directory would have been more robust.
You can use a shell script that you check into your repos that explicitly sets these environment values instead of using the go command directly. As long as you habitually never run "go get" or "go build" directly it ameliorates the issues you mentioned.
Indeed; if you're working with multiple developers and you need these policies enforced, you should use e.g. a makefile. I'm not sure if it's possible to force the use of that over straight `go` commands though.
go env -w allow you to configure your local go installation and set the default value of the config entry when the environment variable is not set. The config entries have the same name as environment variables.
From what I understood from the manual, the configuration stored by "go env -w" is per-machine (actually per-user), not per-repository. So if you checkout on a newly installed machine, and forget to do "go env -w" beforehand (from my experience, people forget even the "git config --global user.email", so it's not unlikely to forget the "go env -w"), you'll be using the defaults.
I agree. I don't like the solution either, but it is what it is. What worries me is that if the envvars aren't set, there's a potential for leaking information.
But, it's still possible to disable these features, so it's not quite as bad as the OP suggested.
> This is the first language in my 23 years programming that reports my usage back to the language authors
I suppose that means you haven't used any of:
1. nodejs, which reports your usage back to npm Inc with the exact same amount of detail (names of dependencies, ip of caller) (and stores much more, since it publishes 'downloads per month', etc)
2. rust, which does the same with crates.io
3. perl, which does that with cpan
4. python, which does that with pypi
5. ... etc
All of those languages have a central registry of packages that has the same level of detail in the metadata it can potentially collect as the go proxy does.
> This "feature" should have an option to disable it.. at the very least.
It does. Multiple options. You can run your own proxy, disable it entirely, or opt out of the go mod experiment and never run go get, but rather vendor by hand with `git clone` or whatever.
> And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow.
Very much a slippery slope. They've communicated clearly how to disable it and what it does, it's in-line with what other language ecosystems do in terms of metadata reported to a central package repository, and I think it's very easy for someone to be okay with this, but to not be okay with any step beyond this line.
> This is different. It’s a proxy in front of others repositories
It's not all that different.
It's perfectly possible to use gems in ruby without ever using the rubygems server by setting every gem to be something like 'gem foo, :source = "https://github.com/foo/bar"'
However, people want to use this centralized repository in the ruby community because it allows for easily updating gems and for convenient caching.
The go proxy is not just a proxy. It also pre-parses out go.mod/go.sum to provide metadata faster and to provide security guarantees.
In a similar way to how centralized repositories are helpful in other languages, the go proxy adds actual legitimate features.
If you look at the public information on its development of this feature, all the motivations cited are about security, performance, and useful features, not information collection.
> This is different
There is a difference. Other repositories, like npm, require someone to opt-in to code being available via it (e.g. type `npm publish`, whether the publisher owns the package or not), while the go proxy default opts-in everyone that publishes a go package publicly and uses the newer go toolchain.
In practice, with how the npm/rubygems/etc communities work, everyone does opt-in if they wish their code to be publicly used, so the difference doesn't seem meaningful to me.
Other than that, the difference seems vanishingly small. In both cases, a 3rd party exists that serves downloads, and it exists to provide performance and discoverability benefits (with the go proxy also does due to supporting version-related operations), and in some cases security benefits.
They're offering a cryptographically signed mirror - the first mainstream one available. You can either run your own personal mirror, run one for your company, or turn it off.
There is a difference between npm, perl, and the others mentioned and Go. Go is Google with a diverse set of products and services. Many people who use Go build competitive services to Google.
What company is using python and building something in competition to the python software foundation? Or something for the others?
This difference is worth taking into account. If Go were part of a software foundation like Python this would be a different story.
You can also pick your own proxy primary and fallbacks.
For example, this should in theory work to make the google-run proxy your fourth choice, with three other proxies attempted before the google-run proxy, and with direct (no proxy) access as the fifth and final fallback:
go env -w GOPROXY=gocenter.io,goproxy.io,goproxy.cn,proxy.golang.org,direct
There is some geographic diversity in that example as well -- I think two mirrors in that list are primarily run in China.
First, I think proxies have their place. Please don't mistake my comment for any issue with proxies themselves.
Second, the don't need Git part is a bit of misdirection. Other package manager (Composer for PHP comes to mind but there are others, too) can pull the source at a version right for GitHub. This is much faster than using Git and people have been doing it for some time. Typically it's pulled down as a tarball.
You don't need a proxy to have git out of the loop for downloads or install time stuff.
Third, when it came to making tradeoffs one was made here. I don't know if it was explicit or not. Protections from the global checksum database as a default were put ahead of privacy as a default.
I wonder what other designs could have kept a certain amount of the global checksum protection while keeping privacy. I came up with a few idea while considering my response here.
Fourth, to really speed up performance in things like CI it's much faster to cache the modules locally. This is something used across programming languages for dependencies and most of the major CI systems have this documented. I looked at using GoCenter months ago but didn't opt for it because caching in CI was so much faster.
We were looking at things holistically rather than just with a single tool.
As I said in the blog post, I appreciate the privacy policy being up front. It is the first link on the proxy website. And, I appreciate the configuration to customize the proxy.
A lot of this has to do with the default. Most people won't change it. A lot of people are going to leak proprietary information to Google without realizing it. Everyone here can decide if that is a good or bad thing for themselves. Hopefully, they will learn the config to customize things if they want.
> You don't need a proxy to have git out of the loop for downloads or install time stuff.
You do with go because go uses git tags for version information and must parse the `go.mod` file at various versions.
Go also supports git repositories hosted in places other than github, where there's no standardized way to fetch a single file at a tag (the go.sum file), or to perform a suitable search for a versoin and its metadata.
> This [pulling github archives] is much faster than using Git
That's only true if you only ever look at a single version of a library. If you have a 20MB git repo at v1.2.3, and ~100KB of files change for the v1.2.4 patch release, using the zip/tar archive means you have to download 40MB total to use it and update it to the new patch. Using the git repository in the first place means the update can only download 20MB since the new `fetch` will be effectively zero data, assuming you cache git stuff well.
I think this is a perfect example of a completely rational implementation that given all the facts almost all developers would agree to - except because of Google's reputation for pervasive tracking, abruptly sunsetting popular products, and general we-know-better approach to software, will become a point of flak for the core Go team.
To be abundantly clear, and because a lot of Google employees read this, the we-know-better comment isn't a jab - because often Google engineers DO know better, but it does occasionally mix with i.e. poor communication and create frustration for the many developers who use Google services and tools.
> This means a proxy, governed by the Google Privacy Policy, is now capturing everyone's module usage by default.
No surprise there, was even called a 'privacy nut' on the neighboring Android 10 thread for highlighting the nature of Google's 'privacy' stance. So essentially they turn on tracking / analytics on by default for module usage without consent or any notice except for the lengthy privacy policy. This reminds me of when Microsoft VC++ embedded analytics in the developer binary when using the compiler. [0]
But then again, Google can do what it pleases since Golang is authored by Googlers.
> "without consent or any notice except for the lengthy privacy policy"
There's also there in release notes:
> "The GOPROXY environment variable may now be set to a comma-separated list of proxy URLs or the special token direct, and its default value is now https://proxy.golang.org "
Had you read the linked release notes, you wouldn't have had to poke around to find that out: it's not some hidden thing. The second paragraph starts off with: As of Go 1.13, the go command by default downloads and authenticates modules using the Go module mirror and Go checksum database run by Google.
It's so that you can have a seamless module version checksum [1]. The checksum in proxy ensures that you don't "trust on first use" whenever you update a module.
The jFrog folks provided a good example with GoCenter. It was optimized for the performance use case. If you used their proxy, released at the end of last year I think, the time to pull a dependency was significantly less than to pull it from GitHub. This leads to things like faster CI builds.
Also, the dependency resolution really just relies on `go.mod` and `go.sum` - having to download the entire git tree just to access different versions of those files is a bit inefficient. (It also makes it harder for 3rd parties to serve different versions of a dependency to different people - and easier for Google to do the same)
The publish field (optional)
The publish field can be used to prevent a package from
being published to a package registry (like crates.io) by mistake,
for instance to keep a package private in a company.
[package]
# ...
publish = false
But I would be curious if someone more knowledgeable than myself would be interested in giving a quick summary of the approach?
Publish isn't necessary for the Go version, since the model is on-demand pull driven. godoc.org is the same way: there's no way for private code to end up there unless your private code is actually public.
The privacy concern is entirely that Google may see the name of some module you attempted to download.
I think they chose a default that was designed to provide a better user experience. If you can't even let the google know what your dependencies / stack is then I think you can run your own as well.
Would be interesting to test the security at some of these companies where this is managements worry. How do these companies worry about dependency leaks but then leave huge S3 / remote exploitable web holes totally open?
If google as an adversary is your worry:
What about all the folks your org emails with that use google in various ways (including perhaps their email stack that they've turned over to google?). How many users are using google single sign on lots of places? How many of their users are using android with telemetry? Browse websites using google chrome? Browse websites that themselves use google analytics.
Google / Apple / AWS / Microsoft are at the root of trust for a lot of what happens online these days.
With different companies there are different agreements. For example, when a company stores something in a private S3 bucket that company has an agreement between it and AWS on who has access to that bucket and for what reasons. The agreements allow companies to know and document risk, SLA, and many other things.
Many companies have not turned their email over to Google. Many companies are in competition or at least coopertition with Google. Many are trying to get a competitive advantage these days.
All of this was a big enough deal for Mozilla that they got a special contract with Google before using Google Analytics that says Google cannot use the data collected on Mozilla sites for other purposes.
Some won't worry or care about the implications of this. There are a bunch who will. It is not an all or nothing thing. It depends on your business if you are building stuff for a business.
Contrary to what someone else has said, this is my least favorite release of Go ever.
I'm salty about how the proxy, which is enabled by default, makes it so much more difficult to pull private modules by having to set environment variables on everyone's individual machines.
Honestly, and I don't think I'm snarking (but would have to reflect a bit on it), "0b", "_", and signed shifts are probably going to make Go more pleasant for me than generics would have. This is my favorite release in years.
It brought one of my favorite parts about writing numbers in languages such as Ruby over to Go. From 10000000 to 10_000_000 is such a readability improvement and should be no-cost.
While I too like the change, there can be a user cost. For example, it makes using grep across a project slightly more annoying. Much like the flexible identifier rules in nim, it feels great but using common tools suddenly needs more thought.
Would it make sense to have an editor display a number in a more readable fashion while the source code itself has the grep-able version? I'm not a fan of ligatures myself but that is also done purely as presentation logic.
Not in Go. Number literals don't have a type until they're assigned to a variable or used in a place that implies a type (e.g. a function argument). For example, the following is valid:
var u uint32 = 10000000000.0 / 100000
even though 10000000000 overflows uint32 and it's also written like a floating-point literal.
Yes in go. `var u = 1e7` gives a float. That means 1e7 is a float literal. You can coerce it to an integer by explicitly typing the target, but it's still a float literal.
You say that, but I find that a lot harder to read (and using `e` notation) than just the number - and readability / clarity is more important than cleverness, especially in Go.
10_000_000 can be read, 5 ＊ (1000＊＊4) and 1e7 has to be decoded. By me anyway, I'm just an average developer that never did well in maths - and you have to write code for the average developer, not for yourself and not for the top 10% smartest people.
Not sure about go, but you can do this in C as a compile-time constant:
#define foo 1.34E8
Maybe go would benefit from a pre-processor? Though it can be somewhat byzantine in C (and result in very unpredictable final source, if abused), it's very helpful to have a complete functional language available for code transformation. It might help go in some cases, too.
I think the question was more around how often is 0b1100 useful outside of a block of consts at the beginning? Surely you're not typing out long constants manually that often?
I like the way Go proceeds as a language, with features added very very slowly and the compiler and tools regularly improving. I hope Rust can settle down into a similar focus soon.
I think Go is the only software release at this point to whose releases I respond with ‘Ooh shiny’ and not ‘Oh god’. I love the way they’re doing this — calm, collected, slow, and with ample heads up.
At this point, I think we all have some JavaScript PTSD.
Yeah, for me the turnaround point for JS was when they added classes - JS is not an oop language and trying to make it look like one doesn't work. Worse, they didn't even implement it properly - no access modifiers.
Call me a bit of a luddite but this is one of the reasons why I'm skeptical about generics in Go - it'll be another language construct to keep in mind. Mind you this opinion of mine is baseless because unfortunately I haven't had the opportunity to do any serious go development yet.
Rust can't settle down right now, because it still has to make necessary changes to critical parts of the language, such as async.
Go always had a very narrow and precise ambition and scope (a better C, aimed at server-side data plumbing). Which is the reason why it was able to nail a set of features from the start and keep it that way.
Rust has a much wider ambition : all the modern languages facilities (generics, metaprogramming, etc) with the speed of C, AND breakthrough memory-safety features.
Go doesn't try to innovate, and i suppose the author see innovation as something you should stay away until it's nothing new anymore.
One day, when it has proper generics, interface validation without compiler error tricks, and an usefull portable UI library, it might be a better Java.
More like a "different Python" than a better one. Way more performant, way more verbose, and (IMO) way better concurrency.
Not to go off on too much of a tangent, but I think asyncio is a huge mess and was a mistake. I still use gevent for everything in Python. https://glyph.twistedmatrix.com/2014/02/unyielding.html is frequently cited as an argument for why goroutines and gevent and green threads suck and why explicit asyncio-style concurrency is better, but I have a lot of issues with this article and find the former two way easier, simpler, and faster to work with.
I went to a Go meetup where I use Go as a complement to Ruby. The speaker and most of the others there were using it as a more expressive/easier C. The middle ground it has carved out seems to let everyone say that their language does something better, but there seems to be a good chance that Go does a reasonable job at what their language doesn't do well (and isn't too far off to learn).
I feel like there is a better way of posting this argument than laughing at them and dismissing what they say, and not providing any reasons why you disagree.
Java got async I/O in 2002, generics and "for (x : iterable)" in 2004, and lambdas in 2014. Vintage 2001 Java was drastically harder to read and slower, though error handling was always less cumbersome than Go.
It sounds like the point the person was trying to make was in use-case, rather than language semantics. As in, where one might've used Java before, they would now use Go. Not that Java and Go are semantically similar.
Installable binaries, around 2000, when the first commercial JDKs started having AOT compilation to native code.
Currently available on PTC, Aicas, IBM, OpenJDK AppCDS (originally from BEA J/Rockit), GradleVM native images, Android ART AOT compilation.
Dynamic heap size, since ever. Every JDK vendor had their own specific switches to configure it.
Versioned modules, since Java 9 alongside Maven/Gradle.
Value types, yeah point taken. There were Azul and IBM specific extensions, ObjectLayout and PackedObjects respectively, and the 2nd experimental release for value types was just made recently available.
If that would be the case, the companies that sell commercial JDKs would be out of business, and yet here they are, selling Java tooling since around 2000.
Maybe you are referring to the "real world" where developers don't pay for their tools.
As for embedding 150MB of JRE / JDK, it is hardly any different than embedding the Go's runtime into every static compiled executable.
Practically speaking this is a non-issue, since its not 1996 anymore. You install the JRE / JDK into your VM or container image. And you basically never think about it until you need to do an upgrade...
I don't know if "all the modern languages facilities" was a goal of Rust from the start. It seemed like the initial goal was to be able to develop a web browser with memory safety.
"all the modern languages facilities" seems to be something that's come along more recently (last couple of years) as a result of a lot of non web browser developers taking an interest in the language.
I feel like Go is the way it is, mostly because its maintainers are also responsible for the continued productivity of thousands of engineers working on Google's numerous Go codebases. Any "breaking change" that requires rewriting Go code is a tangible negative for them, in that it'll interrupt Googlers from getting real work done, and the blame will come back to them.
In a sense, Go is more like a corporate-internal DSL that the public just happen to be able to use. (Erlang is—or at least, was, for the first ten years of its life—another language that is this way.)
Rust doesn't really introduce breaking changes. There is just a lot of cognitive power required to keep up with everything, which can make it hard to read other people's code, or know which is the 'best' way to do a thing, etc. People who disagree with this may simply have more cognitive power/time to devote to keeping up with everything in Rust. I have a day job in an entirely different universe, so keeping up with Rust is hard.
Personally I think this is a pretty positive thing. If a language is at all popular then any breaking change is going to mean work for users, and having those users sitting next to you in the office keeps it fresh in your mind when you're tempted to make that "one small change" to the language.
I hadn’t thought of this before, but I think you are onto something critical about Go.
In a different life, I worked deep in C# and the bowels of the CLR. New versions of C# were both exciting (woo, LINQ, anonymous things, lambdas, and a dozen others each release) and frustrating (ugh, visual studio doesn’t have specialized tools to handle this new stuff).
I’ve been writing Go for 8 or 9 years now and I value the tool chain improvements far more than the language features. Race detection was a godsend. Indeed, even taking away capabilities (or just being more strict) has been valuable—like when multi-threaded access to slices became an error.
In this release, I don’t much care about the new literal prefixes. But I’m excited about the improvement to defer performance and the improved range check panic messages.
In a sense, other languages make it easier to to write new code with each release, but every iteration of Go makes my old code better too.
Rust moves slowly too. I'm not sure where the idea that Rust makes changes willy-nilly came from, but anyone saying that is deeply unfamiliar with the RFC process. It's extremely hard to get new features into Rust.
Maybe Rust moves slowly, but not nearly as slow as Go. I know because I use both languages daily. I spend almost no time in Go code keeping up with deprecations, new APIs and new language features. Not all of those things need to go through the RFC process. For example, I don't think there's any backpressure against deprecating stuff right now. As soon as a replacement comes along, a deprecation pretty routinely occurs just a couple releases (a few months) after it. I don't spend a ton of time in Rust responding to churn, but it's definitely noticeable.
This is why I've been bleating a bit more now than I have before about reducing churn.
N.B. On reflection, perhaps I do have some bias here. My daily work on Go is mostly for $work, and involves one large codebase. My daily work on Rust, however, involves maintenance of dozens of crates. So if there's churn, I feel it repeatedly. So its effect might be magnified for me personally.
One example where Rust is moving faster than Go is the compiler:
* Excluding patch releases, Go has a release twice each year [1], while Rust has a release every six weeks (about 9 times per year) [2].
* For minor (patch) releases (security updates, etc), Rust only cares about the latest version released. Go provides patch releases for the last two releases [3].
* The latest release of the Go compiler (1.13, Sept 2019) is able to bootstrap itself from Go 1.4 [4], released on Dec 10 2014. The latest release of the Rust compiler 1.37.0 (Aug 15 2019) needs Rust 1.36.0 (Jul 04 2019) for bootstrap.
* For the language, I think relative to itself, Rust has considerably slowed down with development compared to the pre-1.0 times. Now it's still evolving more quickly than Go. Rust is getting async_await, const generics, etc while Go is discussing about an error handling operator that Rust already had two revisions of since its 2015 release (try! and ?).
I'm not saying that all of this churn that Rust has is bad. Some of it is good, like the tighter release schedule that prevents half finished stuff to be released. But other things, like the tight version range that the compiler can bootstrap from, could be improved.
Does the compiler bootstrap version matter though? Taking the principle that compilers should compile on the most easily accessible toolchain, writing the compiler in C++ would be the best option of all. But nobody wants to do that.
According to the notes here[1], it sounds like companies care enough about bootstrapping Rust that they are doing it via mrustc. Which seems pretty extreme and suggests they care... a lot. (This was surprising to me to find out, at least, the degree to which they care. I understand why bootstrapping is itself an important ideal.)
Yeah, if you take the principle further it would be even better to write it in C89 because this language is comparatively easy to write compilers for. But the principle is not absolute and it doesn't stand above everything else.
There are major differences in how easy it is to write compilers in C++ vs C89. There are major differences between C++ and Rust with its ADTs and pattern matching. But are there major differences between, say Rust 1.32 and Rust 1.36? I'd argue no.
There are real use cases like distros wanting to maintain/bootstrap a Rust compiler independently from upstream binaries. The more often these people have to port a compiler, the more troublesome it is for them.
Rust seems like it's the odd one out: Swift is written in C++, Go supports bootstrapping from years old compilers, LLVM does so as well, and GCC has probably some of the best backwards compatibility stories out there.
D has been replacing the C++ code with D, .NET made a major reboot with Rosyln where VB.NET and C# got bootstraped (F# was already bootstrapped), OCaml and Haskell have only the runtime in C due to convinience with everything else bootstraped, FreePascal is bootstraped, OpenJDK has the long term goal of replacing C++ with Java/Graal, Jikes was bootstraped in Java, ...
You are misunderstanding me. I don't have problems with the fact that Rust is bootstrapped, but with the high rate this bootstrapping is happening. That a 12 week old compiler is already considered as too old to compile the newest rustc. This seems wrong.
Perhaps it's the number of things changing that leads to the feeling of rapid movement. After all, 60 1mph increases are all tiny by themselves.
Not sure if that's true or not though. Rust is my daily driver for the last year and even coming from Go (after 6 years) I don't feel the pace is hard to keep up with, or that my code breaks frequently.
In fact, I don't think my code has broke as a result of changes once since I switched to Rust. It has happened .. twice maybe, via dependencies, though.
Rust needs to settle down to reach broader adaption, because the ecosystem must stabelize. No business oriented project manager would choose Rust as an alterative to Go or Java while Rust has "only" a community http lib on version 0.12.
Sure, async needs to be shipped and polished, but then Rust needs to tell the world: "We have all you need and as stable as you need."
Heh, I like the way Rust proceeds as a language, with regular releases and improvements. I hope Go can make quality of life improvements with similar speed.
> The GO111MODULE environment variable continues to default to auto, but the auto setting now activates the module-aware mode of the go command whenever the current working directory contains, or is below a directory containing, a go.mod file — even if the current directory is within GOPATH/src. This change simplifies the migration of existing code within GOPATH/src and the ongoing maintenance of module-aware packages alongside non-module-aware importers.
Oh wonderful! This makes switching back and forth between module projects and 'legacy' projects far easier. (Well, at least for me - since I tend to keep all cloned git repos inside the GOPATH. I don't have a problem, really!)
I occasionally use Python for large-scale processing of structured data (XML, JSON) and the data's really messy as you'd expect (malformed responses, missing fields, normalisation, etc..). One of the things putting me off of Go for these kinds of projects is it's much more verbose/harder to handle edge cases - it feels like I'm less productive writing the program, but end up building something much more stable than the Python equivalent.
I'm definitely going to see if Go is acceptable for my next project since I really need the performance for my workload, and I've wanted to put this great language to work in something that isn't a toy project.
> is it's much more verbose/harder to handle edge cases - it feels like I'm less productive writing the program, but end up building something much more stable than the Python equivalent
This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone.
> I really need the performance for my workload
Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.
Go's not perfect in this respect, either. I had your typical map[string][]interface{}, because the data in the map was an array of records, but different records in different cases, and made an error in an append(), so ended up with nested arrays. That's correct according to the type definitions, but unwanted. Generic could have helped here. So I think the language still needs to grow a bit.
> This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone.
Yep. Maybe I just suck at programming but I couldn't write enough tests to avoid not being fed up debugging stack traces in production. I started using Python's type hints, since this was the source of most of the bugs which helped but I'd rather just use a strongly typed language next time.
> Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.
I've got a few million records pulled from an API so I'm extremely I/O bound and goroutines are easier to work with than Python concurrency. I've also got reasonably complex data processing beyond just XML/JSON parsing.
I think on the whole Go is still going to be faster, but thanks for the heads up.
Have you tried async? I find it much easier personally to reason about and debug than channels, and is perfect for IO-bound problems. If you really like channels, note that Python 3.9 will have sub-interpreters and channels as well: https://hackernoon.com/has-the-python-gil-been-slain-9440d28...
Goroutines are quite a lot nicer than async for a few reasons. Firstly, there are no “forgot to await” errors, secondly there is no need to worry about some library making a sync call way down the call stack and therefore blocking your event loop, and thirdly goroutines can use all cores on a CPU to parallelize CPU intensive tasks, which means you don’t block the event loop unless you really are out of CPU. Blocking the event loop is a really big deal since it can cause health checks to fail which can cascale onto other instances and bring your app down. We’ve experienced this with both CPU- and IO-bound event loop blockages, and they’re really hard to debug. These just aren’t issues that happen in Go.
A lot of what you describe is for CPU bound problems, most of the problems I face personally and the the problem, in particular, the parent mentioned are IO-bound. Async in Python is a near-perfect fit for this kind of problem space and is really easy to reason about and debug as you can step through and debug one line of execution at a time.
Regexps is another area where GoLang used to lag behind Python (for certain types of expressions?). Not sure if things have changed, but this issue is still open 4 years later: https://github.com/golang/go/issues/11646
Not sure I fully understand the NFA vs DFA/re2 choices -- seem like the issue above conflicts with the comment below. Can someone clarify?
From [1]: "Go has chosen to follow the re2 path (not surprising, since Russ Cox is a major author of both Go and re2). re2 has much better performance characteristics than some other regexp engines, in that it never has an exponential slowdown, but that comes at a cost for other regexps (https://swtch.com/~rsc/regexp/)."
> This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone.
Luckily, you don't have to choose between those two outcomes:
Python's optional type hints even with mypy are nowhere near strict/expressive than a statically typed language like Go. Type hints in Python are an afterthought and it shows in a lot of cases.
Type hints in Python are IMO great and quite expressive!
You can create sum types (a bit verbose but they are there).
You can introspect them at run time and create things like pydantic[0] and fastapi[1].
Strict null checks with Optional[T] are great!
Now I'm not too familiar with Go's type system, but with protocols, Unions, Optional[T] and other generics offered by MyPy[2] I'd wager the type system is __more__ expressive than Go.
The type system is more expressive, but much clunkier and also it needs to be expressive to accommodate all of the weird things people do in Python (and you still can’t model SQLAlchemy types).
> Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.
Using encoding/json maybe but also using 3rd party libraries? jsoniter is pretty fast, I just ran some tests and for my use cases it outperformed Python.
I kind of wondered that myself. All the completely automatic marshalling using tags seemed like it assumes things are very nice (which they aren't when working with some of the terrible data sources I've seen). Everything else feels very cool to use Go though, I just used it on my first real project.
Yes I do actually -- my best advice is to watch this course. It's hands down the best course I've ever seen (across any language, actually), and he jumps RIGHT into it, almost from a computer science aspect.
He doesn't talk about learning the language at all (he assumes you know all that), instead he fills in all the gaps in a very thorough way that are left out from all the tutorials and such you'll read. I've read dozens and dozens of Go articles while learning and I would say 85% of those 15 hours was NEW information to me, that's how good it is.
There are 2 versions, I watched the older version because he's more lively in it, however I have heard the newer version is better because he does cover a lot more up to date tooling. I went and rewatched the section on debugging and structs on the new version.
For example, its 15 hour course, and in the literal first 5 minutes of the first coding video, he talks about struct field memory alignment on different CPU architectures. That's when I refilled my coffee and said 'dis gon be good'.
Aside from that, I would say pick a good project structure that works well, invest time time into researching the different ways (also covered in that course)
I also use go modules which are new, and make things way easier as far as packages.
Definitely make sure your editor has gofmt and goimports set to auto-run on save, those are lifesavers. Goimports is smart enough to look across packages and in your own package, which is cool
All the other suggestions about actually writing code is basically covered entirely in the ultimate go course (when to choose pointers vs structs, how to structure code, good error handling practices, etc)
I agree with your sentiment. I still reach for Python (and Pandas) for a quick and dirty, get-it-done program and then when feasible I move over to Go.
I am slowly ending up with a library of utils from doing this sort of work. In the end it does end up more stable (and more verbose, which is OK with me) and easier to maintain. I've said for a while Python makes me productive and Go makes my whole team productive.
I actually just started to use go on windows just to check it out. First impressions from a newbie:
- Getting started with go and vscode is kinda bad right now. I installed go, set up GOPATH, installed all kinds of extensions in vscode and then, somewhere down the official tutorial I learn about GO111MODULES, go mod init, vendor folders and the language server... I was completely confused and still am. I removed all extensions except the main one from vscode and the language server was crashing like crazy until I unset GOPATH. Works now, mostly. Intellisense and code formatting seems to be still unstable now and then. But geez, that first experience was horrible.
- The syntax is very weird after writing too much C#. Sometimes I have the feeling, that declarations are reversed just for the sake of it.
- Why does go need := with the colon? I know it's a declaration, but I haven't figured out the reason for the colon.. Seems weird..
- I love gofmt and format-on-save. But it was not the default setting. Still don't know if I should use another one.
- I hate configuring vscode. After hearing only good things about it, the configurations and keybindings are horrible and that I need an extension for each and every small thing. Maybe I'm in the minority, but Visual Studio is miles ahead in this regard, I think.
- Fast compilation times are awesome. I love, that I can create a simple exe without a framework dependency.
- I can parse 2GB of simple JSON files in less than 20s. I'm awed.
- I have no idea how to compile some modules like the sqlite one. Seems to need gcc..? Do I have to fully switch to Mingw/WSL?
- I was losing my mind when I saw my program giving different results on each run... Until I found out about that randomized map iteration.. Funny.
- Finding an item in a slice: Yeah, sometimes generics would be nice but let's not start a holy war...
- Need to get used to error handling.. It can get really verbose and those scoping rules are strange.. I miss exceptions, but I can get used to the go way.
- I love that I can use the official docs locally: godoc -http=localhost:6060
- I had to use google and stackoverflow more than I initially thought for a 'simple' language... But well, I guess that's normal?
Sorry for the rambling. I love the simplicity of the language and hope to get into it more. But the initial experience is still, umm, not streamlined, it seems? But I hope that's just temporary.
Wow, this is really mind-opening. Everything makes sense now. Thank you.
It's also funny, that they stuck to the * pointer notation which led to the problem, where they could not tell apart a pointer from a multiplication, so they put it to the left of the variable name. It's the small things...
But I guess * is just too convenient and ingrained.
The weirdness is that ":=" allows redeclaration (within a scope) but only partial, whereas `var` completely forbids redeclaration.
That is:
var a, b = f()
var a, b = f() // NO
var a, b = f()
a, b := f() // NO
// note that the second declaration only partially overlaps with the first
var a, b = f()
var a, c = f() // NO
var a, b = f()
a, c := f() // yes
I think the point is to make sure you know what you are doing. That declaring a new variable should always be an intentional act, and this makes assignments to typoed variable names error rather than just creating a new variable.
You will if you ever do concurrency stuff in go. The way goroutines capture their working variables in a closure make things like `x := x` useful and clear (once you know why you do it in the first place )
if err := fn(); err != nil {
}
// the result of fn does not affect the identifier err
// in the enclosing scope
you're scoping err just to that if block. that's ... just about the most common phrase in all of Go. it won't affect any err defined prior and it won't exist outside of the if block.
You'd have to do this to make the value stick around or affect the enclosing block:
var err error
if err = fn(); err != nil {
}
// err has the result of fn here
I am trying to help a Windows-using team ramp-up on Go dev, and it's pretty rough.
Currently my advice to them is ditch VS Code and download GoLand, the editor change is jarring but the experience is much more fluid on Windows overall. Also transparent support for modules, per project.
For compiling when your code (or a package you're using, e.g. sqlite) has a cgo dependency - afaik YES you will need to go down the dark road of cygwin/Mingw, if you want your development cycle to be at all normal... and even then, my observation is compilation times are an order of magnitude slower on Windows :(
At GopherCon I met folks who worked for huge, "old guard" Windows companies who said "just run a linux VM" to develop in Go.
If you just need to compile for windows once and a while (e.g. for releases only, not as part of your normal dev cycle), check out xgo [1], which uses docker to deal with all the cgo mess. No cygwin/Mingw to install/worry about.
Best of luck! If anyone else has tips for working in Windows, I'd also love to hear them.
Personal recommendation, if you need to stay on windows and don't want GoLand - set it up to use wsl 'remotely' [0]. All the offline ability of a local environment, but with the software compatibility (and ecosystem) of linux. (That being said, GoLand has been great on both Windows and Mac for me - the only thing VSCode does that GoLand doesn't is remote dev, and that's supposedly coming soon™ [1])
Thanks for your feedback. GoLand looks really nice (like everything from JetBrains), but the price is a bit steep for just playing around once in a while. But I'll keep it in mind.
I'll try the WSL Route and see how it goes. (VSCode already made me install that extension.)
> - The syntax is very weird after writing too much C#. Sometimes I have the feeling, that declarations are reversed just for the sake of it.
It's really C and languages directly derived from it which are the odd ones out. "postfix type" ordering dates back to at least Pascal, and is very common especially but not solely amongst languages with type inference.
Just look at what more direct C derivatives have to do to back-fill type inference: invent some weird-ass pseudo-type to take the type's place. I also think it maps better to how I read / write / think. "f is an integer", the definition of f is more relevant than its type.
> - I can parse 2GB of simple JSON files in less than 20s. I'm awed.
100MB/s doesn't seem that impressive? rapidjson or serde can do several times that, simdjson reaches 2GB/s.
> - Why does go need := with the colon? I know it's a declaration, but I haven't figured out the reason for the colon.. Seems weird..
Without the : it means every assignment might be a declaration so you need a "declaration inference" system, and every language which does declaration inference has odd tradeoffs and corner cases e.g. implicit declaration in the local-most scope (Python), implicit method-wise non-local (ruby), implicit global (javascript), …
Explicit declaration is a very good decision I think. As far as I'm concerned it's one of Python's big annoyances (and javascript but there it's easy to lint away as you also have explicit declarations, you can just ban implicit ones).
> Your explanation makes sense. It would complicate the compiler in this regard.
The compiler part's easy, it's the meatpiler which gets into an odd funk when implicit declarations don't do the expected thing (which will eventually happen). Not having implicit declarations is much simpler and more straightforward for everyone involved.
> Why does go need := with the colon? I know it's a declaration, but I haven't figured out the reason for the colon.. Seems weird..
I remember Rob Pike ever said that one if the reasons of needing := is to avoid declaring many error variables in a function, such as err0, err1, err2, ..., and at the same time we don't need declare then as uninitialized variables.
> Getting started with go and vscode is kinda bad right now. I installed go, set up GOPATH, installed all kinds of extensions in vscode and then, somewhere down the official tutorial I learn about GO111MODULES, go mod init, vendor folders and the language server... I was completely confused and still am. I removed all extensions except the main one from vscode and the language server was crashing like crazy until I unset GOPATH. Works now, mostly. Intellisense and code formatting seems to be still unstable now and then. But geez, that first experience was horrible.
Yeah, there are a lot of reasons for this. Things used to be easy when it was just GOPATH, but that didn't support every use case, so modules were added. Now the advice beginners get is fragmented and confusing to navigate. Also, the error messages for modules are terrible. These are tractable problems, but it's definitely a rough time to be a beginner (which is unfortunate because Go is otherwise very beginner-friendly).
VS Code is a different product altogether and its Go integration is pretty buggy in my experience, especially its handling of modules. Go really needs a bullet-proof (free) editor story. If you want a premium IDE (a la Visual Studio), I've heard good things about Goland.
> Need to get used to error handling.. It can get really verbose and those scoping rules are strange.. I miss exceptions, but I can get used to the go way.
The scoping rules are pretty straightforward and standard. I'm guessing "shadowing" is tripping you up. Error handling is verbose, but this is a good thing IMO; it's really clear what is happening and there is no special control flow for certain data that is arbitrarily classified as an error. The real issues with errors are the lack of stack traces and other standard structures for programmatically inspecting / manipulating errors. Some of these issues are being addressed in go1.13 but it will take a while for the best practices to be established and propagated through the community.
> Sorry for the rambling. I love the simplicity of the language and hope to get into it more. But the initial experience is still, umm, not streamlined, it seems? But I hope that's just temporary.
I think it will be. You're entering at a time when there's some churn in the best practices and it makes for a confusing time to be a newbie. Hang in there!
Thank you for your elaborate perspective. I initially thought that I'm just dumb, but I guess there are really alot of new things coming around lately, so I'm ok with some minor problems.
Really hope to use GO more and that those rough edges will be gone after a while.
The way I view it is that Go is for people who write software while Rust is for those who like to think about how to write software.
Perhaps I’m old and grumpy but I feel that at this point I’ve had every language discussion before and if people start to moan about lack of metaprogramming support I just leave.
I have yet to see a project that failed due to lack of high order abstractions
I’ve worked on two projects (in early x86 assembler) that ran out of steam because they took too long to do and it was hard to onboard new developers.
I’ve also abandoned a few paths during projects because of libraries that were overly abstract, didn’t work and were thus hard to understand and maintain.
A product at a company I used to work for had to reimplement a huge chunk of code because the original designer left and the metaprogramming behemoth he wrote was too hard to understand, extend and debug. That one chunk of code cost about $500k and 6 months to replace. I’m unsure of the total cost of that effort as it delayed the next release at a critical time.
Abstraction levels need to be appropriate. Enough to make things simple - not so much that it makes life harder.
I would have to look at this since I’m not familiar with the codebase, but I’ll have a look. Thanks!
I’m not quite sure this is what I meant from the description of the problem. This looks more like a problem of trying to program Java in Go. What I was thinking of was more along the lines of modeling things in ways that either disconnect you from the problem you are trying to solve or solve it with too much abstract cruft grafted onto the model so just dealing with the abstractions themselves becomes more work than a more direct and naive approach.
> The way I view it is that Go is for people who write software while Rust is for those who like to think about how to write software.
More accurately speaking, Rust forces people writing software in a specified way. You need to think about how to write software much less in Rust than in Go. In Go, you really need to think more to choose a better solution from many potentials.
Anecdata: he Rust people I know do a lot of talking. They have very little running in production. The Go people I know do less talking and tend to ship a lot more code into production. Or more precisely: more value per month.
And mind you: I manage quite a few projects where Rust would be a great fit because it would replace C/C++, but for embedded projects Rust just isn’t getting the traction it needs to convince key industry players.
1. Cross-platform
2. Fairly mature ecosystem
3. Pretty easy to pick up coming from either a dynamic or static language background
4. Somewhat simpler approach to concurrency (although I've created my share of race conditions)
5. Backed and used by Google (if that is positive for you as a developer)
6. Is a pretty good multi-purpose language
I think in comparison to (my limited understanding of) Rust, Go is beginner friendly. Also, I think Rust has most of the above covered as well, but it seems like there are lower level primitives with functional aspects built-in. Haven't needed these in Go, so haven't investigated them there.
Applying the bug fixes in a dependency I'm using when a new version of it is published… [1]
Sarcasm apart, I wasn't talking about features. But cargo is a really well-designed package manager. I've been using Rust full-time for 2 years now and it had never annoyed even once.
[1] the sarcasm was referring to the Minimum Version Selection algorithm used by go's package manager, which is a really bad case of NIH from Go's teams who decided to throw away a thriving community work to do their own stuff, ending up with something different from what everybody else has been for more than 20 years. And unsurprisingly, it's really bad…
Would it be fair to say that Go's concurrency support makes it a good match in the network services space? I see Go more often in Docker and cloud contexts.
Personal opinion: no. Go's concurrency is hard to control or monitor, and it's hard to make higher level abstractions that are both safe and convenient. It's of course possible, but the tradeoffs are rather severe.
Go's concurrency shines best in short-lived single-purpose processes, which is a near perfect fit for CLI tools. When you don't care if something gets abandoned or fails to make progress and can just ctrl-c, the relative simplicity is 100% worth it. Other parts of Go work well here too, especially the very low startup overhead (unlike, say, python). For long lived processes though, like most services, it feels very error-prone or very boilerplatey and manual.
> Go's concurrency shines best in short-lived single-purpose processes, which is a near perfect fit for CLI tools
This is not reasonable. Go's concurrency shines for its flexibility. It has nothing related to whether or not the written programs are short/long-lived. It is the only language which gives me a happy and fun experience in concurrent programming.
> For long lived processes though, like most services, it feels very error-prone or very boilerplatey and manual.
This is not reasonable too. Whether or not a written program is robust depends on the experience of the authors. Rust may reduce many mistakes at compile time, but it can't prevent all mistakes. Programmers still need experience to avoid many mistakes.
That's kinda like saying "language ergonomics and api design don't matter, you just need to be careful".
Which is true to some degree, you can build anything in anything if you're careful enough. But programming history is full of repeated, preventable errors, and languages where X error is impossible or much less likely if you're less than perfect.
Go's rather far down "unsafe" in my books, especially in regards to concurrency and error handling. For simple code, it's pleasantly simple. For larger, robust code, it's several times more verbose than other langs I've used, and often requires continually re-implementing common "wheels" (that should not usually be written by hand, as they're finnicky to get right, which is also easy to miss in code review) to get around a lack of generics.
Go's concurrency features provide a larger super set of possibilities of other language. If you only do concurrency programming in a simple way (just as most languages), it is hard to make some mistakes even if you are a moderate Go programmer. On the other hand, Go allows you to do concurrency programming with many creative ideas which are hard to apply in other languages. Surely, you should be an experienced Go programmer to implement these creative ideas.
Rust does prevent some errors at compile time, but achieving this has its own cost, such as sacrificing flexibility and simplicity. For many use cases, the sacrifices are not worth it.
> it's several times more verbose than other langs I've used, and often requires continually re-implementing common "wheels"
If the common wheels need generics to implement, I have nothing to defend. I can only say generics will come to Go eventually.
> Could you expand on what you think is hard about the concurrency in Go?
Go's concurrency is cheap[0] multithreading (shared-memory concurrency). It has a tagline that you should not share memory ("Do not communicate by sharing memory; instead, share memory by communicating"), however nothing prevents sharing memory (so it will eventually leak in) and channels are pretty slow[1] so it's not uncommon to go back to regular shared-memory primitives (with no safety net).
You've got no idea what your application state really is (even less so once things start falling over), and there are no built-in tools for reliability e.g. you can't get notified that a goroutine has keeled over or anything unless you implement that yourself.
> And perhaps give an example of a language that makes it easy/easier?
Erlang / elixir was built specifically with that in mind. It uses lightweight shared-nothing concurrency (non-OS processes) and the system builds-in ways for processes to have oversight over others (e.g. monitors & supervisors) or get killed alongside them (links) and provides behaviours which bundle known patterns, etc…
[0] as in "not expensive", as in uses quite a bit less memory & especially kernel resources than OS threads or processes[1]
[1] though this point should not be too overstated, people commonly misuse "the C stack is XMB" with "creating a thread consumes XMB of physical memory", a C stack is mostly a regular allocation and thus usually just VSS until actually exercised
Not the OP, but if you’ve ever had to write 100% fault tolerant concurrent Go code, you find yourself having to add a lot of monitor-recover-restart “boilerplate” that ends up being quite complex, depending on the task at hand.
It helps if you follow certain principles like making your concurrent tasks idempotent etc., but Go doesn’t force you to write things that way, so it takes a bit of remembering each time.
If the task is relatively simple, you end up writing a lot of concurrency infra for a little bit of concurrency, which can become frustrating if you’re doing it for the Nth time.
It sounds like a problem that’s ripe for abstraction, but it turns out that it’s quite hard to build abstractions that get all the trade offs just right for each case. There’s some worker pool style libraries out there, but if you care about performance you can usually eke more out by writing a custom solution.
As for alternatives, I have heard that erlang has a better story for this, because of the inherent restrictions of the language, but I haven’t used it much.
In my experience, Go gives you enough rope to make this task “merely annoying” (rather than impossible or astoundingly challenging) without giving you enough to truly hang yourself with. You can get really tangled up though :)
Rust has broadly the same concurrency support, with a more powerful compile-time race detector (but one that also comes with a learning curve). The main difference is that Go uses M:N threading, while Rust uses 1:1 threading with optional explicit async/await constructs.
Rust is really great, but I feel like you are overselling it a bit here:
- async/await is available in the nightly version of Rust (1.39), but not in the stable version (1.37).
- 1:1 threading is not strictly equivalent to M:N threading (the main practical difference being that the stack size per OS thread is larger than the stack size per goroutine).
Writing that "Rust has broadly the same concurrency support" is misleading right now. But it could be true in a near future.
The features that each language offers are the same: threads, channels, and blocking I/O (though Rust has more features for safe concurrency--Rust prevents data races statically, while Go is not even memory safe
in the presence of such races). What is different is the performance characteristics. In some cases, M:N will be more efficient; in some cases, 1:1 will be. But just as I wouldn't say Go is lacking FFI features because M:N makes cgo slow, I wouldn't say Rust is lacking concurrency features.
Rust I/O are strictly similar to Go when using 1:1 OS threading, but become conceptually different when using async/await.
> Rust has more features for safe concurrency--Rust prevents data races statically, while Go is not even memory safe in the presence of such races
Agreed. That's a big advantage of Rust.
> What is different is the performance characteristics.
Performance is a feature. It's significant enough to justify using one language or the other depending on the project at hand.
> In some cases, M:N will be more efficient; in some cases, 1:1 will be.
The main advantage of the M:N model, compared to the 1:1 model, is the memory usage, because each goroutine starts with a small stack (a few kB). It makes possible to start a larger number of M:N goroutines than 1:1 threads.
> But just as I wouldn't say Go is lacking FFI features because M:N makes cgo slow, I wouldn't say Rust is lacking concurrency features.
Go's FFI works but is slow. It's a well known fact. Rust's concurrency story is not stabilized yet (areweasyncyet.rs). It's a fact too. I don't see a problem with acknowledging both :)
For the threading difference, do you mean that Rust needs 'normal' threads plus async/await support, but Go can just lean on its built in concurrency support for both cases? i.e. Go threads instead of explicit async style code.
Your sentence implies that Go green threads can do both of what 1:1 threading and async/await can, but it's more complex than that: there's a tradeoff between the simplicity (only one concurrency primitive) and the capability of the said primitive:
- with it's M:N model, Go cannot really do FFI efficiently, while both 1:1 async/await have no problem with that in Rust.
- goroutines are cooperatively scheduled, while OS thread are preemptively scheduled. If you have hot loops, Go's model won't be able yo guarantee fairness between goroutines, which may be a problem depending on your usage. OS threads don't have this problem.
- with goroutines you won't achieve the level of performance you can reach with async/await.
Go and Rust fills a different niche of programming and take a different stand on this tradeoff: Rust gives you more power at a complexity cost. Go offers you more simplicity, with less power.
The reason cgo is (relatively) slow is small stacks, not M:N threading.
Small stacks (and M:N threading) are needed to efficiently implement tens of thousands of goroutines.
CGO is slow mostly because it needs to switch to a larger stack when calling a C function.
It's a trade-off.
A different Go implementation could make Goroutines map 1:1 to threads and have fast cgo calls at the expense of slow goroutines.
Rust is not exempt from those trade-offs. They chose fast FFI and smaller runtime. They paid with slow threads.
Async/await promises to be the best of both worlds but it comes at a cost of great complexity, both for the programmer and the implementor.
At the end of the day under the covers it's just threads that need to be managed in a complex and often invisible way plus a complex rewrite of your straightforward code into a mess of a state machine.
I can confidently say that learning to use goroutines took 10x less time than learning async/await in C#.
I understand goroutines better than I ever understood async/away.
> Small stacks (and M:N threading) are needed to efficiently implement tens of thousands of goroutines.
Tens of thousands of threads are no problem with 1:1 threading on Linux.
> They paid with slow threads.
I would not call 1:1 threads "slow threads". If they were slow, then the 1:1 NPTL would have not defeated the M:N NGPT back in the day when this was being debated in the open source OS community.
> complex rewrite of your straightforward code into a mess of a state machine.
The entire point of async/await is that you don't have to do a complex rewrite of your straightforward code. It remains straightforward, and the compiler does the transformation for you.
Given the context, it's not a memory problem was implied, because small stacks are not actually relevant, because C stacks are not committed upfront (at least on unices, not sure about linux). So your threads are more likely to take up a page each (plus kernel datastructure overhead) than the 8MB allowed to a C stack.
> I can confidently say that learning to use goroutines took 10x less time than learning async/await in C#.
It's a matter of preferences, I've always preferred async/await to threads because you don't need to juggle with channels and you're way less likely to cause a deadlock.
> it comes at a cost of great complexity, both for the programmer and the implementor.
Is that really harder to implement than a M:N runtime like Go's?
> The reason cgo is (relatively) slow is small stacks, not M:N threading.
You're picking nits here. Without small stacks there would be no point using M:N threading in Go…
And Cgo is not only relatively slow: most of the time it is slow enough to destroy all performance benefit of calling a FFI function. It's a pretty big deal.
Rust has no backward compatibility, a very disputable feature with language revisions when compiling, and it is rare for you to have a large package from a stable branch without loading the package from the test branch. So in its present form it looks interesting, but it is absolutely not adapted to use outside of pet projects.
Nice to see that out-of-bounds panics will now include the offending index. Was that difficult to implement, or just controversial? Seems like something that could have been added a long time ago and saved much frustration when debugging.
Strings are fixed-size, though. A string is a pointer and a length.
It's true that you can't use a slice as a map key, though, even though slices are also fixed-size (pointer, length, and capacity). As I recall, the argument is that the equality rules for slices are unclear. (maps and funcs can't be used as keys either, for the same reason: they're not comparable.) Is a 0-length slice equal to a nil slice? Is a slice with len 3, cap 3 equal to a slice with len 3, cap 4?
In practice, this is rarely a problem. It most commonly comes up when you want to use []byte as a key, in which case you can convert the []byte to a string. This overhead of this conversion (specifically when it occurs in the context of a map key) has been optimized away for a few versions now.
Maybe "fixed size" was the wrong way to phrase it. What I meant is that a string points to an unlimited amount of data and all of that unlimited amount of data is taken into account when deciding equality of strings. And my question should have been, is string the only datatype with that feature?
If it is, it means Go is like Lua, where if you want to use any compound data structure as a key in a map, you either marshall it to a string or you implement your own hashtable.
In Python you are allowed to use tuples of immutable objects as keys, for example. Much nicer than serializing everything to strings.
As for "in practice this is rarely a problem", I agree, but prefer languages where the standard library doesn't have these arbitrary limitations. For example I might want to implement multivariate sparse polynomials as maps from exponent tuples to coefficients (the polynomial 3 x^2 y^3 z + 7 x y z^4 corresponds to the map {(2,3,1): 3, (1,1,4): 7}). That's straightforward in Python but awkward in Go (in Go I might be tempted to just set a maximum number of variables and use arrays or exponents as keys).
Oh, cool! This still doesn't quite get rid of my complaint since the size of an array must be known at compile time, but it's good that they don't all have to be the same size.
Would it be too hard to just compare arbitrary slices and arrays based on the contents of said data structures?
Converting arbitrary bytes to strings should fail at least some times, since not all byte sequences are valid UTF-8, and afaiu, in Go, the string type should always contain valid UTF-8.
Go strings do not need to contain valid UTF-8, although in practice they usually do. A 'range' loop over a string will loop over its UTF-8 codepoints, but in general, a string can contain arbitrary binary data. See https://golang.org/pkg/builtin/#string
I'm not sure what you're talking about. Map keys in Go can be strings, numbers, pointers, some structs, etc. And that was the case since at least Go 1.0.
I always have trouble explaining this limitation of Go maps.
First, a string can point to an unlimited amount of data (i.e., it can be of arbitrary length) and all of its characters are taken into account when deciding string equality.
Now, an array type in Go is not like that: the length of the array is part of the type. There is (or at least there wasn't when I last checked) no type of all arrays of ints, say, but there is a type of array of 5 ints, a type of arrays of 8 ints, etc. Each of those is of limited size.
A slice is like a string, it can point to an arbitrary amoun of data, but slices are not allowed as map keys.
A struct has a fixed number of members and is only allowed as a map key if all of its members are of types allowed as maps keys.
So a struct type that is allowed as a map key type can only point to unlimited data all used for equality if it contains directly or indirectly a string somewhere. Among types allowed as keys, the only source or arbitrary-sizedness are strings.
For example, if you want to represent multivariate polynomials sparsely as a map from exponent vector to coefficients (i.e., the polynomial 3x^2y^3-7xy^4 corresponds to the map {(2,3)=>3, (1,4)=>-7}) you can only do it in Go with the built-in maps awkwardly. You could pick a maximum number of variables and use arrays of exponents as keys. You can leave the number of variables arbitrary and convert exponent vectors to strings (i.e., use {"2,3"=>3, "1,4"=>-7}). I find it frustrating that those are my options.
I see now, thanks. Existing maps will probably remain that way forever, but the new Generics Draft Design[1], if implemented, should allow people to easily write a wrapper that will work on anything.
Fixing the existing maps sounds a lot easier than adding generics. All that would be required is to allow slices as keys. If there is a concern about immutability you could just say mutating a slice used as a key leads to undefined behavior.
AFAIK, “undefined behaviour” is not a thing in Go The Language (there is some undefined behaviour in the Go The StdLib). The Go Specification[1] doesn't even contain the word “undefined”, and only has two instances of “unspecified”.
I guess that if immutable slices became a thing, one would probably be able to use them as map keys. But so far the Go Team is very reluctant about adding immutability to the language.
Lots of currently allowed map keys can be mutated such as structs or arrays. Let whatever happens when you mutate them (what does happen?) happen also for slices.
Uhm, no, they cannot be mutated. If you use a structure with two integers as a key and then change one of these integers, then you have a new value, and a new key, just like with integers.
That didn't quite clear up for me what happens when you mutate keys, so I ran a quick experiment. It seems like maybe the map stores a copy of the key, as the map doesn't "see the mutation".
package main
import "fmt"
func main() {
x := [2]int{1, 2}
m := make(map[[2]int]int)
m[x] = 5
fmt.Println(m[x])
x[1] = 3
fmt.Println(m[x], m[[2]int{1, 2}], m[[2]int{1, 3}])
for k, v := range m {
fmt.Println(k, v)
}
}
This prints:
5
0 5 0
[1 2] 5
I'd be perfectly happy if slices were allowed as keys with the same semantics as arrays...
Is there some information about how to set-up your own proxy/checksum DB?
EDIT: I understand that there are environment variables you can set, but how do I go about deploying my own proxy/checksum server? Is the source code behind https://sum.golang.org/ published somewhere?
The documentation makes clear, if someone accidentally commits something they shouldn't (i.e. a file containing your users personal details, credentials, etc...) then it is stuck in their proxy forever. It is clear that google employees believe we live in a perfect world where all employees never make mistakes.
> "...the mirror aims to cache content in order to avoid breaking builds for people that depend on your package, so this bad release may still be available in the mirror even if it is not available at the origin. The same situation applies if you delete your entire repository. We suggest creating a new version and encouraging people to use that one instead."
If you commit and push your credentials, they have been compromised, full stop. This proxy should make no difference to how you handle such a compromise.
Actually it does make a difference. If an employee accidentally commits a CSV file with customer private data, are you actually suggesting that it should not be possible to remove it?
I disagree. The quicker that the personal data can be taken down, the less chance there is for someone to discover that personal data. I can't imagine any large company being comfortable with developers choosing technology that makes this harder to deal with than it should be.
(I am not at all suggesting this scenario is ok, simply asking how one would deal with it if the scenario should occur.)
The go checksum database / go sum / sum.golang.org appends your module name, version, and hash of it's contents into a permanent, immutable, public log as soon the first time it's seen. But sum.golang.org doesn't host any code.
On the other hand, proxy.golang.org re-hosts your module, and you can know that the proxy isn't serving you maliciously or differently than anyone else by verifying it via the public checksum database at sum.golang.org.
I don't believe proxy.golang.org, or any proxy for that matter, is required to host all of the modules and versions listed. It's a dumb caching proxy, and must deal with all of the real world complexities of hosting user content, including taking it down if deemed necessary. So if they need to take something down from the proxy they can, and the checksum database will only permanently retain the hash of the content that makes up the module.
Hmm, you definitely have a point. I was thinking more API tokens and private keys, but confidential data itself falls into a separate category, it cannot be revoked or rolled.
https://codeengineered.com/blog/2019/go-mod-proxy-psa/