Yeeeah, this made me lose half a day learning about and then unwinding “toolchain” being accidentally set across all our microservices…
Made a little harder to track down initially as we have a replace directive pointing to a local proto module in each service. It was here that toolchain was set which forced it to keep appearing in the top level module.
Should’ve been opt in behaviour by default not enforced.
We’ve decided to sit on n-1 of Go and I’ve already had an example last month of a well used OSS library setting toolchain to 1.22 as a patch release (which could be considered a breaking change) which fortunately was quickly reverted.
> Many people believe the go line in the go.mod file specifies which Go toolchain to use. This proposal would correct this widely held misunderstanding by making it reality.
That doesn't sound like a good reason to automatically download binaries and run them.
Is it difficult to update or install a new version of Go and are there frequent updates in Go spec introducing new features that it is necessary to auto install the compiler itself ?
Supply chain attacks are on rise and not a new concept and yet we see these changes.
This is not the first time Go lang has introduced a questionable opt-out feature [1]. They backed out but looks like there were no takeaways from that episode.
The binaries are fully reproducible from source[1] and are published in a transparency log to provide assurance that your go command is downloading the same binary as every other go command. This is state-of-the-art in preventing supply chain attacks and better than every other language.
Are you really surprised that people who explicitly refused to learn from 50 years of computer science and software engineering history are refusing to learn from their own mistakes, or even acknowledge them as such?
I'll be raising an issue with my distribution of choice to disable this behavior by default. They're the last line of defense for users' security and privacy, and we as an industry have been trying hard to circumvent them. Maybe precisely because of that reason, who knows.
I'm not sure I like this feature. I prefer to be in control of the software that runs on my system, without any automated downloads. But I'm a nixos user so...
But strangely I liked this feature, mostly because it allows me to use newer features without needing to wait nixpkgs to update the Go compiler to the version 1.23.
It is easy to disable if you don't like though, something like `environment.sessionVariables.GOTOOLCHAIN = "local"` or `home.sessionVariables.GOTOOLCHAIN = "local"` should do the trick.
Setting the GOTOOLCHAIN environment variable to local disables this. I don't use nix so I'm not sure if they set it by default but you can with set it with `go env`
It forces what the foundation wants you to do. It takes away control from you, it's unethical.
Not only that, it opens a can of worms. A compiler should be a compiler.
Not an self updating application, nor an dependency modulator from GitHub. How can I trust it when it does all these things?
Call me old fashioned at 35.
If you want the latest then go download the latest. Is that now to hard for the user?
Just because the latest is out doesn't mean it's any better than the previous version. What happens in a CrowdStrike scenario? What happens when Go gets retired in 50 years?
I don't want to work with the latest. Should I? TCL 9 is getting there but TCL 8.7 is still perfectly operatable. Should I be using 9 because it exists? My work only has 8.6 on production.
So your toolchain updates and they've removed a thing. You've got to hunt down the previous version, let alone needing to discover why it was working yesterday and not today. Unnecessary overhead.
You use a dependency that's not updated for the future version?
What stops someone from crafting a malicious binary? Malware hijacking the download path?
Auto-updating takes away your integrity. Your making blind trust that everything is what it is.
How can I be sure that the updated compiler is the compiler and not a malicious crafted version? If you can't trust the compiler how can you trust your code?
Yes, I could turn it off, but could I turn it on instead.
I shouldn't need to turn it off, I'll update when I want to update tyvm.
The way to look at it: The Go toolchain is just another dependency to your project, declared in go.mod, just like the other modules, and handled accordingly (download, verification etc.)
You are confusing the `go` toolchain and the binaries generated with Go toolchain.
The `go` toolchain itself is compiled statically, this is why this works without issues even in NixOS. Binaries built with the toolchain may or may not be linked against libc depending if you enable or disable CGO. But this is not related to the toolchain, that will work regardless since the toolchain itself is statically compiled.
Edit: or in other words, the `go` toolchain is built with CGO disabled.
If I understand correctly, Rust acts similarly: if you have a `rust-toolchain.toml` file in your repo, any usages of `cargo` will automatically install and use that version of Rust. https://rust-lang.github.io/rustup/overrides.html
It seems like that will change in the (near) future according to the following github issue[0]. A quote from one of the developers, rami3l, in that thread[1]:
> My current plan is indeed to remove implicit installations entirely.
Oh sad! Dang I actually really liked the feature, it's super convenient for keeping developer environments in sync. I left a comment in that thread asking for clarification.
>> Oh sad! Dang I actually really liked the feature, it's super convenient for keeping developer environments in sync. I left a comment in that thread asking for clarification.
Here is a slightly contrived, but realistic example of why it is a bad idea:
1) Attacker discovers vulnerability in an older version of the Rust toolchain
2) Attacker creates useful crate and helps it to get widely adopted or becomes trusted contributor to a crate that is already popular
3) Attacker creates and publishes crate changes with exploit code and rust-toolchain.toml to trigger use of older, vulnerable Rust toolchain
4) Unsuspecting developers build the trapped crate or something that depends on it and get owned
Installing toolchains automatically without the user's consent or permission is a supply chain attack in waiting for both Rust and Go.
Perhaps they could make it a configuration setting that developers could opt-in? That would let developers who want automatic toolchain installs to have it and others who do not want it (or whose employers will not allow it) to not have it.
In Go case though the version can only go higher, not lower (e.g.: it will not download a toolchain if the Go version is set to lower than your current one, only higher). So I can't see the same attack being executed here.
>> In Go case though the version can only go higher, not lower
That is good to know, assuming that the newer hypothetical toolchain is not vulnerable (e.g. a zero-day in the newer toolchain).
My opinion is still that toolchains (newer or older) should not be implicitly installed without the developer's explicit permission. This could be a configuration setting that the developer has opted-in or a "This package requires toolchain version X. Install it? (y/n)" prompt.
> That is good to know, assuming that the newer hypothetical toolchain is not vulnerable (e.g. a zero-day in the newer toolchain).
This would still be really difficult to explore:
0. An undiscovered zero-day in the newest version of Go
1. The only way to force users to use the newest toolchain is if your project has dependencies where the attacker has control, so e.g.: they can change their go.mod and set a `go` directive that upgrades to the later version
2. You need to update to this new version, because this will make Go complain that your `go` directive inside your `go.mod` is out-of-date
3. After all this, yes, Go will download the newest version of Go (that is vulnerable)
But I would argue if the attacker already has 2, they have better ways to attack you (e.g.: they can add code to `init()` that will run once the module is imported, and this could be explored once you build and run the binary).
Again, not saying that this is an impossible scenario, just find it highly unlikely.
> This could be a configuration setting that the developer has opted-in or a "This package requires toolchain version X. Install it? (y/n)" prompt.
I concur at this part though, however keep in mind that this `toolchain` feature is more akin to adding a module dependency, since it reuses the whole module system.
I think there is already lots of trust that is implicit when you are managing modules, and this new feature doesn't make it worse. If anything, considering that the binaries can only come from golang.org/toolchain and I assume Go already check the checksum of every module that it downloads (and also that Go is perfectly reproducible: https://go.dev/blog/rebuild), if anything this is probably more trusted than a random module that you add to your project.
Thanks, I understand this; but what I don't understand is, wouldn't it be easier for the same attacker to do the same thing by exploiting a vulnerability in a different crate, and include that other crate as a dependency?
As for configuration: to me, having it be opt-in negates the entire benefit. My point is that automatically installing the correct toolchain makes it far easier to collaborate with others who aren't nearly as obsessive about Rust as I am.
>> wouldn't it be easier for the same attacker to do the same thing by exploiting a vulnerability in a different crate, and include that other crate as a dependency?
Possibly, which is why the example is a bit contrived. In most cases, the toolchains will likely be more trusted and be on approved lists whereas binaries created by third-party crates are not.
For more secure environments, explicitness is valued and automatic installation of anything is frowned upon because it can introduce unvetted changes which could include vulnerabilities.
It depends on what work is being done and how much toolchains and ecosystem can be trusted.
>> That seems like a lot of hoops to jump through considering that rust allows arbitrary code execution during compile time anyway.
If you mean build.rs build scripts, yes, those do run, but it is not arbitrary code. You can view and inspect them before building. If you need more security, you can download all the dependencies and build inside an isolated container.
You can't post like this here, so I've banned the account.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
Automatic updates are always a good idea... until they aren't ! It's an open door to problems ranging from subtle version semantic difference (try to find why what was working yesterday is not working today without any change) to hacking.
In corporate environment, it might trigger some unexpected alarms (or be blocked)
Gentoo got this right: by default, the "old" behaviour should be kept (no automatic update) and it should only be opt-in
What i'd want is a way to revert back, if the user sees a problem.
The automatic updating toolchain is not a bad idea, and in most cases it works great. Once in a while though, there's a fuck up and breaks your build or something changes that you didnt expect.
The best way forward is to keep the old tool in a backup location, and have the user be able to revert the update with a single command. Then the user is able to safely return to the old one when the inevitable breakage happens. You could even opt-in to analytics for this action, and the owner of the tool would know if a particular update is a screwup (aka, many people reverted it).
To be clear, the new toolchain doesn't replace the old toolchain, but is installed alongside it, and is only used for modules that explicitly request the newer toolchain in their go.mod file.
If you have a `go.mod` that sets a `go` directive with a version bigger than the version your Go toolchain is currently running, it will download the binary automatically and pass the commands to the new binary. However this doesn't replace your `go` binary in say, PATH, the new toolchain will be downloaded at `GOPATH`, the same as it does with your modules. This is why there is nothing to backup here, nothing is deleted.
Also, keep in mind that if Go is downloading a new toolchain, it means your project will not work with the current toolchain: because if you set `go 1.23` in your `go.mod`, and your current `go` is version 1.22, it would fail regardless until you either fix your `go.mod` or upgrade your `go` toolchain. What Go is doing here is the later, automatically for you.
If you didn't commit the breaking change to the repo, couldn't you just revert your local changes? Trashing the mod and sum file and running "tidy" should take care of the rest, I would think. Maybe I'm missing something?
> By the way, this only works well because Go binaries are static, one of the things that make the language reasonable good.
Only by default, as the Go toolchain supports dynamic linking for ages, and without third party dependencies called via cgo, or stuff like DNS resolution on Linux.
What I meant is that the toolchain itself is linked statically, this is why it can be just downloaded from the internet and it will work without extra setup from the user.
Yes, Go will build the libraries once you call `go build`, and yes, they may link to the system C libraries for things like you said. But the `go` binary itself is static.
I'm broadly a Go fan, but this strikes me as a terrible decision. Critical tools should not be working through magic like this. Just fail with a clear error and let the user fix it as they see fit.
If they want magic like this, it should be opt-in, not opt-out.
This doesn't make sense. The toolchain will be updated only IF you say your project only runs in Go >= XX. This is what `go` directive in `go.mod` says: my projects doesn't work in versions of Go older than this. This is why it happened to me, I declared I need `go 1.23` because of range-over-func, and the toolchain respected my wishes.
This is not an "enforcing you to use the latest Go version, even if you don't wish for it".
Python 3 is not backwards compatible with Python 2 so why would it do that?
Maybe you meant "imagine if Python 3.6 automatically updated itself to Python 3.12"... Which would be amazing! Except that Python 3.12 is also not backwards compatible with Python 3.6. There's no saving Python.
Even in that case they did the behaviour change properly:
> To ensure backwards compatibility with existing code, the new semantics will only apply in packages contained in modules that declare go 1.22 or later in their go.mod files.
CMake and Rust and Android and lots of other systems have mechanisms like this that allow you to introduce opt-in breaking changes, but unsurprisingly Python doesn't. They just decided it was ok to start breaking backwards compatibility.
That's still among the strongest backwards compatibility guarantees in any major language. Maybe Java would be close? It's certainly better than C++ which is famed for backwards compatibility, and light years ahead of Python.
I don't like Python much, but they will definitely not pull off something like this (again), because they've actually learned from that mistake and broadly agree that it was one.
It’s more like if virtualenv looked in requirements.txt for the Python version and used that version locally. Other languages like Haskell and Scala have been doing that sort of thing for ages.
Made a little harder to track down initially as we have a replace directive pointing to a local proto module in each service. It was here that toolchain was set which forced it to keep appearing in the top level module.
Should’ve been opt in behaviour by default not enforced.
We’ve decided to sit on n-1 of Go and I’ve already had an example last month of a well used OSS library setting toolchain to 1.22 as a patch release (which could be considered a breaking change) which fortunately was quickly reverted.