Very cool! I've been watching the "toolchains in Wasm" landscape for a while, and seeing a Clang/LLVM toolchain running in Wasm is awesome!
YoWASP has also had an LLVM toolchain working in Wasm for a while too[1], although it seems like this version solves the subprocess problem by providing an implementation of `posix_spawn` whereas the YoWASP one uses some patches to avoid subprocesses altogether
My biggest question marks around this version are about runtime/platform support. As I understand it, this toolchain uses WASIX, which (AFAICT) works with Wasmer's own runtime and with a browser shim, but with none of the other runtimes. Are there plans to get WASIX more widely adopted across more runtimes, or to get WASIX caught up to the latest WASI standard (preview2)? Or maybe even better, bring the missing features from WASIX to mainline WASI like `posix_spawn`[2]? I'd love to be able to adopt this toolchain, but it doesn't seem like WASIX support has really caught on across the other runtimes
I believe Deno's LSP does use this push model, the missing ingredient is that the LSP server itself is implemented in Rust, but bridges to TypeScript's Language Service API in JavaScript/TypeScript. The TypeScript side calls functions like `readFile()` that Deno then needs to provide, so they need to get the in-memory file contents from the Rust side to the JS side.
I think garn is really cool! If I were still in the Nix ecosystem, garn would probably be my go-to
Building something on Nix obviously has a ton of benefits: well-proven tooling, lots of mindshare already, TONS of packages. Everyone knows that Nix-the-language is fairly unintuitive for beginners, and tools like garn, Devenv, Devbox, etc. all seem to be trying to let folks use Nix without exposing them to Nix-the-language. I think this is a good goal to have and helps make Nix more accessible
But I really thought it was worth exploring the Nix-free design space too. For example, I think derivations in Nix are... kinda complicated and weird! Having a fixed set of phases specifically designed for autotools builds always seemed a little weird to me. By ditching Nix, I was able to see what the world looks like when you ditch the idea of phases completely, for example
Another interesting one was ditching the absolute path for the store path. Getting everything working by resolving relative paths was surprisingly tricky, but the end result is that you don't need to be root to install Brioche, and you can also take a Brioche-built binary (along with the resource dir it outputs) and ship it to another Linux machine and it'll just work. That's only something you can do if you see what happens if you explore outside the Nix space
So I'm hoping Brioche can _at least_ be an interesting case study in how you can take a lot of Nix's ideas and get a lot of the same value without having to take Nix's approach directly (or who knows-- maybe some of those ideas end up back in Nix someday!)
I'd definitely like to have good performance even for large build graphs! I'm hoping the laziness exists "where it counts". To walk through an example, if you build your backend, and your backend calls the function `postgres()`, and that calls `openssl()`, and THAT calls `gcc()`, etc., etc., each function is basically building an object to represent its chunk of the build graph (each function returns a "recipe"). Nothing gets built until that object gets returned from the top-level function and the runtime does something with it
In other words, the eager part is basically constructing the build graph. Maybe I'm wrong but I don't that this would necessarily be slower than the lazy version. In practice the most complex build graph I've made is basically the full chain of Linux From Scratch builds (that's the basis for my toolchain currently), and I think that takes about 400-500ms to evaluate. It's about 160 build steps, so it's not _simple_ but I know build graphs can also get a lot more complex, so I'll just have to keep an eye on performance as I start to get into more and more complex builds
Maybe I'm missing something but intuitively I'd expect this approach to be fairly efficient-- as long as build scripts only call these functions when they're used as part of the build graph
I think it really depends on your definition of "large". I don't think strict eval + full build graph can scale to something the size of nixpkgs, for example.
I mentioned in another comment that this is why Bazel uses simple strings to form dependencies on other targets. That way Bazel can manage the laziness and only evaluate what is needed without needing to use or invent a language with lazy evaluation.
But that is also the big downside (in my opinion) - the full build graph necessarily can't exist purely in starlark (at least for Google-scale projects) which increases complexity of the tool overall.
Edit: I'd like to add, though, that I think it's perfectly fine to not scale to Google scale or nixpkgs scale! Many many projects could still benefit from a great build tool.
Honestly, I think the "stringly-typed targets" thing isn't too bad, having used Buck2 quite a bit, and being a Nix user for 10+ years. If anything, it's a small price to pay for some of the other smart things you get in return, like the far more fine-grained action graph and the tooling around BUILD files like querying. One weird benefit of that stringly-typed bit is that the BUILD files you have don't even have to meaningfully evaluate or even parse correctly, so you can still build other subsets of the tree even when things are broken; at ridiculous-scale it's nearly impossible to guarantee that, and it's something Nix does worse IMO since evaluation of the full nixpkgs tree is slow as hell in my experience but a requirement because a single eval error in the tree stops you dead in your tracks.
Also, no matter how much I might not like it as a language nerd, I think Starlark is simply far more "familiar" for your-average-bear than the Nix language is, which matters quite a bit? It might be more complex in some dimension, but the problem space is fundamentally complex I think. So other factors like how approachable the language is matters. (And at least in Buck2, you can use MyPy style typing annotations, thank God.)
> One weird benefit of that stringly-typed bit is that the BUILD files you have don't even have to meaningfully evaluate or even parse correctly, so you can still build other subsets of the tree even when things are broken; at ridiculous-scale it's nearly impossible to guarantee that, and it's something Nix does worse IMO since evaluation of the full nixpkgs tree is slow as hell in my experience but a requirement because a single eval error in the tree stops you dead in your tracks.
I think you get more or less the same property with Nix. You can have all kinds of errors, even certain syntax errors in the same file, but if they are unneeded for the current evaluation, they won't cause any problems.
As for language familiarity/approachability - this will always be a matter of opinion, but I personally don't think it makes sense to optimize for the casual contributor. Plenty of people know python, but I never see casuals making anything besides trivial changes to bazel build files. I don't think they gain anything by familiarity with python, they could very well copy paste nix or any other language. And if they get in to trouble, they will call in the experts.
Nushell has definitely been on my radar for a while! but I missed DetSys was playing around with it, I'll have to give that a look
One neat thing about Brioche is that Bash isn't really baked into it the same way as the Nix ecosystem. In a lot of the docs and examples, I used the `std.runBash` Brioche function for running scripts, but that's a fairly simple 11-line function[^1]. I wanted to make it just as easy to add `runZsh`, `runFish`, `runNushell`, etc. functions that work just as well as Bash (and they could either be put in the std library or in their own packages). So hopefully, there will be a Brioche `runNushell` function in the near future :)
I've been asked about overrides and attributes a lot! That was one of the sacrifices I had to make to go with a more traditional language, so it's definitely a negative point compared to Nix. That said, I'm hoping to have some conventions and features that will cover _most_ of the use-cases that overrides give you, but that's definitely still future work at this point and I probably won't be able to cover everything overrides do
And yep, I think having a "Brioche for Nix users" guide makes a lot of sense, although it's not the first time that question so I'll probably stand up a first-pass version of it sooner rather than later (my Nix skills are also pretty rusty-- I'll need to brush up a bit first before I write it!)
ooh, porkg looks pretty interesting, will definitely keep an eye on the project!
Definitely agreed about the lockfile ideas! I went with a fairly similar design, although I haven't really escaped the "rebuild the world" situation yet and I've rebuilt a _lot_ of copies of gcc from source by now! (that's also partially because I set up the packages repo as a workspace[1], which I felt was easier to iterate on in the early days... I might eventually split each package into their own projects or separate repos so they can keep lockfiles independent of each other)
and yeah, I definitely felt the temptation to write my own DSL but I stayed strong! I just knew that it'd be a huge uphill battle, especially because I wanted to provide good editor support (IMO implementing a language is (relatively) easy, but implementing a language with good error messages and LSP support is crazy hard)
> What are your plans for cross-compilation or heavy package customization? One of nixpkgs coolest party tricks imo is that you can just change the stdenv and get a musl static binary or cross-compiled binary.
So in general, I don't think I'm going to have anything quite as powerful as Nix's overrides. But I'm hoping most of the use-cases for it will be covered by some simpler options:
- Since build definitions are functions, package authors can just take arguments for the things they want downstream users to be able to customize (e.g. `configure` flags, optional features and plugins, etc.)
- I haven't built it yet, but I think adding support for dependency overrides would be fairly easy, a la Cargo. Basically, you'd just fork or clone the package you want to tweak, make your tweaks, then set an "overrides" option to use it instead. I know that's not a super satisfying answer, but that should help cover a lot of use cases
- For toolchains specifically, I have an idea in mind for this as well (also not implemented at all). At a high level, the idea is that packages could use "dynamic bindings", which you can then override for downstream recipes (this would require some new runtime features in Brioche itself). The toolchain itself would effectively be a dynamic binding, letting you pick a different recipe (so you could swap glibc for musl, or gcc for clang, etc). Cross-compilation would also be built on this same feature
This was one of my bigger pain points with Nix as well: there was a lot of "reinventing the world" just to avoid network access. With Brioche, I stuck with disabling network access by default, but there's an escape hatch to specifically opt-in to networking (by calling `.unsafe({ networking: true })` on a process recipe). My thoughts are that Cargo, NPM, Poetry, etc. have all done a great job building amazing tooling to download and verify resources from the network against a lockfile, and I wanted to be able to leverage as much of that as possible. So, for example, `npm clean-install` will give you more-or-less the same guarantees that Nix does, so my thought was as long as it's encapsulated properly, I'd rather lean on existing tooling (that's also why I used the term "unsafe", you need to make sure you only access the network with tools that do sufficient verification!)
I've generally stayed away from using the term "reproducible build" when talking about Brioche, because I don't feel like it fits the reproducible-builds.org definition (though I don't think Nix does either). But, if a build is cached locally or in the registry, then you're guaranteed to get the same result, since it'll just re-use the cache
The sandboxing also gives pretty strong guarantees around hermetic builds[1]. So I think you could do reproducible builds _within_ Brioche (and I'd like to add tools to make this even easier), but I'd say Brioche itself doesn't give you reproducible builds out of the box
That's actually really great to hear! I might need to play with this tonight then.
It looks like the project files aren't radically dissimilar to Flakes, so I think you're really on the right track for making something that could be really useful for a lot of people. Great work!
If the underlying package eco system is a mess then it’ll also be a mess in Nix. To this day Nix doesn’t really have a good answer to Python or JavaScript packaging.
If the underlying package manager is good (Rust, Go) then there’s just no need to reinvent the wheel.
I interpreted the parent to mean that it might not be fair to assume existing package managers have done a _great_ job at downloading and verifying, especially verifying, resources from the network. There are businesses that exist attempting to solve this problem like socket.io. Safely installing the correct dependencies for a project is still not a guarantee from any of the major package managers.
Another YouTuber beat him to the punch about 2 years ago actually! Here's the moment from his stream where he finally reached the Farlands [volume warning]: https://youtu.be/VAvQ_kT73W4?t=25696 (at the 7:08:17 mark)
YoWASP has also had an LLVM toolchain working in Wasm for a while too[1], although it seems like this version solves the subprocess problem by providing an implementation of `posix_spawn` whereas the YoWASP one uses some patches to avoid subprocesses altogether
My biggest question marks around this version are about runtime/platform support. As I understand it, this toolchain uses WASIX, which (AFAICT) works with Wasmer's own runtime and with a browser shim, but with none of the other runtimes. Are there plans to get WASIX more widely adopted across more runtimes, or to get WASIX caught up to the latest WASI standard (preview2)? Or maybe even better, bring the missing features from WASIX to mainline WASI like `posix_spawn`[2]? I'd love to be able to adopt this toolchain, but it doesn't seem like WASIX support has really caught on across the other runtimes
[1]: https://discourse.llvm.org/t/rfc-building-llvm-for-webassemb... [2]: https://github.com/WebAssembly/WASI/issues/414
reply