Hacker News new | past | comments | ask | show | jobs | submit login
NixCon Live (nixcon.org)
147 points by FRidh 5 days ago | hide | past | favorite | 48 comments





I hope Nix takes off for one reason. I don't want people to do same builds again and again especially for Rust it's painful.

I want to install binaries with cargo instead of doing the same build again and again. Similar to https://cachix.org

I don't honestly know why that isn't the case yet, perhaps storing binaries takes too much storage that Mozilla doesn't have money for. Or perhaps it can't do guarantees like the ones Nix gives to allow binaries being shared of builds.

This is the main reason that makes interfacing with Rust as a plain user a pain.


I think another good reason is that it lets you specify language-agnostic dependencies, including build tools etc. If Nix really takes off, we can stop dealing with language-specific package managers.

The situation for installing haskell programs with nix is great, practically everything from hackage is available in the binary cache.

Is this harder to achieve for the Rust ecosystem (due to some properties of Haskell perhaps)? Or just no one put in the effort in doing it?

Is it expensive to maintain all the binaries for all of Hackage?


Is this harder to achieve for the Rust ecosystem (due to some properties of Haskell perhaps)?

You can actually do this today. The buildRustCrate function in nixpkgs creates a derivation for a single Rust crate. You can combine this with crate2nix [1] to make derivations for a crate and all its dependencies in a Cargo.lock file. Every compiled crate will be a separate output in the Nix store and you can upload them to a binary cache such as Cachix.

This is what I have been using for CI in many of my Rust projects in combination with Cachix to get very fast builds.

[1] https://github.com/kolloch/crate2nix


There's a couple of challenges here with Rust. None are insurmountable, but there are just other things that are higher priority.

There are 15,000 packages on Hackage, but 48,000 on crates.io. According to http://www.modulecounts.com/ hackage is getting about four new packages a day right now, and crates.io is getting 54. This means that expense is more of an issue.

I don't know about Haskell, but in Rust, many projects tweak various compiler flags, for performance, or whatever. You'd need to get the exact flags the same too. It's possible there's a long tail effect here.

https://doc.rust-lang.org/nightly/rustc/platform-support.htm... is way longer than https://gitlab.haskell.org/ghc/ghc/-/wikis/platforms (I am not 100% sure the latter is canonical). This isn't a hard stopper, and there's probably a long tail effect here, but it's still a factor in you getting a precompiled binary or not.

Rust relies more heavily on inlining than Haskell does, and relies heavily on monomorphization This means more work to do during compilation; it also means that even for a "precompiled binary", you still end up needing to do a lot of work on the final compilation.

There are existing tools like sccache that you can use to implement a binary cache, so people that really want this feature also have the ability to get it, and some do. It's just not a globally available thing.

If you look at Rust's compile times, generally, even once your dependencies are compiled, it's still slower than we'd like. You only compile dependencies once, and so, while binary libraries may help your initial build, they don't really affect later builds at all, and that's most builds. So more effort is going into improving that, rather than focusing on pre-compiled dependencies.

That's off the top of my head...


If Rust does more inlining than Haskell, it does _a lot_ of inlining! Haskell doesn't monomorphise (it calls it specialise) much by default though.

FWIW, that is mostly gut feeling, I don't have good data on that. I probably should have softened that sentence. And it's not really that Haskell is missing some sort of optimization here, it's that these things all tie together: Rust tends to monomorphize a lot, which produces more code in general, including stuff to get inlined. It's one of the big challenges to get things to compile quickly; they're sort of inherently in tension with one another.

It's not inherent to the language. It's inherent in the build tooling. Back in the cabal days haskell had the same problems. Nix is a more hermetic build and packaging tool which is why it's better at it.

Very curious to read about what it means to have a `hermetic build and packaging tool`.

In my naive view, there should be a way to at least cache the latest successful build from a package version. Then when users `cargo install ..`, it will check if a build was already made and retrieve it.


"at least cache the latest successful build from a package version"

Tools like guix and nix aim to describe packages in such a way that for the same input (package dependencies) you get the same output.

If the output is a function of the input, and you know a cached build of a package was built with the same input, then using the cached build of a package would be the same as if you'd built it yourself.

It does depend on whether the packages are written in a "pure" way. (Or "impure", such that building with the same inputs might result in different outputs). https://github.com/NixOS/nix/issues/2270


I think that 'hermetic' here means that the build cannot depend on anything other than its input - it's built in a sealed environment where nothing other than the inputs is available. This property makes it more likely that the build products do indeed depend only on their explicit dependencies and not implicit dependencies.

The problem with caching just on package name and version is that there are many different possible flavours. For example, multiple different versions of the sourcecode may have the same given version number. Things such as target architecture, optimisation level, and optional features confuse things further. Nix therefore uses a hash of all the inputs as the cache key.


Nix is a tool for deployment, that happens to have some build capabilities. This is how I view it anyway.

I don't need nix for C and C++, and several other compiled languages.

What is missing is cargo support for native libraries.


I find Nix incredibly useful for C and C++ development, though. It allows me to get a working development environment (including completion support) for any major open source C/C++ project by simply placing a default.nix or shell.nix file with the following content and running nix-shell:

    (import <nixpkgs> {}).your-favorite-project

Been using nix package manager on Ubuntu, so far the experience has been mixed, while the package management and nixpkgs is great but the tooling and documentation around the nix language are lacking.

It's one of those tools that are technically miles ahead of their competitors but struggle to gain traction because of beginners facing little cuts and bruises in using them.

That's why I'm interested in the Nix Modules proposal to be presented in the conference that aims to bring better usability to the language with types and docs.

Also the updated website is a lot easier to navigate now and explains the use cases of nix quite well


Nix (specifically nix package manager on MacOS) sits in a weird spot where it is absolutely indispensable for me personally but I am absolutely hesitant to recommend it to others without extreme caveats. I really wish the UX of it were much better, which ideally then would lead to higher adoption and less frequent 'osx-isms'.

I've been using it on my main computer for about 6 months now.

It's been the steepest learning curve, but yeilded the biggest dividends so far.

The other day I needed to install some software from a specific 3 year old branch. Copying the default.nix for the up to date version and changing a few of the options got it up and running with minimal fuss.

There are certainly some rough edges around usability, but the fundamental value prop is immense.

Also just using nix as a secondary package manager on mac or some other Linux distro is great, I'd expect shell.nix to start showing up in an increasing amount of OSS projects. It showed up in hashicorp's latest project


I've been using Nix OS since this year and am mostly happy with it, but if I had to name my largest issue with it then it's that updates are so traffic intensive. Basically, glibc is linked dynamically on most distros, including Nix OS. But if there is a glibc update that only changes the .so file, it still updates all packages depending on glibc.

A more expansive list of my issues with nix (with some curse words :p): https://gist.github.com/est31/005f17628d55fda27b401d3c994233...


I'm confused with your first point?

> No concept of source packages. There is only a binary cache (hydra) and the derivations in nixpkgs.

NixOS _Only_ has a concept of source packages. The cache is a (sound) optimisation om top of it. just disable the binary cache and you can build all packages from source.


Nix packages don't contain any source code.

The package definition describes how to fetch the source code from a source (like a Git repo or a hosted archive) and build it. The built result only contains what is necessary at runtime.

A sizeable amount of packages don't even fetch source code but a prebuilt binary which is then fixed up to work with Nix.

There is a source cache, but it is optional.

As an example, check out ripgrep [1]. It uses `fetchFromGithub` to retrieve the code.

[1] https://github.com/NixOS/nixpkgs/blob/3bb54189b0c8132752fff3...


Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

True, which is absolutely sufficient for most use cases.

I'm currently doing some work for ML and data science companies where full reproducibility and introspection is very much desired.

So you need to run your own source cache to provide that guarantee, because you can't count on cache.nixos.org still providing the source code from a package built 4 years ago.

But that's why I love the IPFS cache efforts. [1] Running your own node to pin all required sources should then be relatively easy.

[1] https://blog.ipfs.io/2020-09-08-nix-ipfs-milestone-1/


Software Heritage is also helpful here, and Guix is integrating with it - see e.g. https://guix.gnu.org/blog/2019/connecting-reproducible-deplo...

Nix is also integrating with the Software Heritage:

https://www.tweag.io/blog/2020-06-18-software-heritage/


Debian's archive reaches back 15 years now: https://snapshot.debian.org/

It also contains source code.


Are those sources target independent or specific for each new build of the package? That is, is there a new source code package on hydra when one of the dependencies changes? Or does it only change if the package itself changes?

Also, they are only available when hydra builds the package anyways, right? So if some package is not built by hydra (like how it used to be for the texlive packages), it'll still download the sources from the various places they are hosted.

As for the hash, it's good that the source code is hashed, but my main concern was that it was downloading from external sources in the first place. This is bad for privacy, as those hosts know I'm downloading from them, as well as for reliability, because the hosts might not have as good uptime as a debian package mirror.


A sibling comment replied to your first paragraph, so just about the second two:

>Also, they are only available when hydra builds the package anyways, right? So if some package is not built by hydra (like how it used to be for the texlive packages), it'll still download the sources from the various places they are hosted.

Yes.

>As for the hash, it's good that the source code is hashed, but my main concern was that it was downloading from external sources in the first place. This is bad for privacy, as those hosts know I'm downloading from them, as well as for reliability, because the hosts might not have as good uptime as a debian package mirror.

That's a true and valid concern, but note that it's the same situation as with Debian: If the package is built upstream by Debian/the NixOS Hydra instance, then you have reliable, private access to its source code so you can rebuild it. If it's not built/packaged upstream, then you need to get the source from somewhere else.

The discrepancy is just that there's packages in Nixpkgs which are not built upstream, and which get built only locally on your machine or your own Hydra instance. There are not many of these, but yeah, it would be nice to fully get rid of them.

Or, an interesting option would be to build the source for more packages on Hydra, without actually building the binary for the package. That wouldn't be too hard, if someone adds an expression for doing it.


> That's a true and valid concern, but note that it's the same situation as with Debian

Good point!

> an interesting option would be to build the source for more packages on Hydra, without actually building the binary for the package. That wouldn't be too hard, if someone adds an expression for doing it.

Yes, that would be awesome!


Are those sources target independent or specific for each new build of the package? That is, is there a new source code package on hydra when one of the dependencies changes? Or does it only change if the package itself changes?

Fixed-output derivations are used for sources (they are content-addressed in the store), so the latter.


Binary packages make me sad too and I wish there were a way to mark them as such using a `meta` key.

It starts to become a bit of a grey area in some cases though. For instance - java packages. Is a .jar a binary? Probably. But so many java applications rely on pulling loads of .jars down from maven. Are we going to sit down and figure out how to build all those jars from source? It's not uncommon for there to be literally hundreds.


I find that binary packages are quite rare for free/open source software. So avoiding proprietary software (with, e.g., allowUnfree = false) gets most of the way there.

I agree that it would be nice to tag (with meta) FOSS packages that aren't built from source, though. Every instance of that is a bug, IMO...


> I find that binary packages are quite rare for free/open source software.

Unfortunately this isn’t the case with some languages that have their own package managers, the prime example being Java as the parent commenter mentioned. It’s near impossible to build Java applications without fetching tons of binary jars from maven that Debian just gives up on providing their own package in many cases[1]. While Nix does build Java applications from source, the dependencies are fetched from maven in binary form.

[1]: https://wiki.debian.org/Hadoop


Next talk starts at 15 UTC, that is in 2 hours.

I'll be speaking how about how to get Nix to become mainstream.


Oh, that's good to know because cfp.nixcon.org says 15:00 but without timezone, so I assumed it was 15:00 in my timezone!

https://cfp.nixcon.org/nixcon2020/schedule/


Yes, blame me for this one. I didn't have things ready quickly enough to fix certain important user experience aspects like "showing the timezone".... which is kind of a basic and critical need.

Anyways, the widget at https://2020.nixcon.org/#program localizes to your timezone (but also shows weird dates sometimes).

Times from the cfp subdomain are all in UTC.



> To watch the stream from VLC on your computer, run:

> nix-shell --packages vlc --run "vlc https://dash.nixcon.net/dash/master.m3u8"

I thought this was pretty cool, but unfortunately, this doesn't work on macOS...


Won't really save you today, but you should be able to replace vlc with mpv; I'm currently viewing the presentations from macOS in this manner.

Thanks! This worked:

    nix-shell --packages mpv --run "mpv https://dash.nixcon.net/dash/master.m3u8"

That command temporarily installs vlc and launches vlc from the command line. If you already have vlc installed, you can do "vlc https://dash.nixcon.net/dash/master.m3u8"

Yes, I know. nix-shell on macOS works fine for me, but vlc fails to install because the Nix package is not supported on macOS.

I was just sad that I couldn't use the cool Nix magic to view the stream.


It's important to note that Nix(OS) is really 3 things:

A functional language, a package manager and an operating system.

I switched all my desktops and laptops to NixOS at the beginning of the year, and I'm starting to use introduce it at multiple companies for data science / ML work environments with full reproducibility and Kubernetes bare metal deployments.

Main benefits:

* pure build system

* extremely easy way to set up and switch between custom environments with their own sets of dependencies (nix-env, nix-shell)

* nix (the package manager) is cross-platform and works on MacOS and other Linux distros (although that's not always smooth sailing...)

* Ephemeral environments that are quickly set up and torn down, like the `nix-shell --packages vlc --run "vlc https://dash.nixcon.net/dash/master.m3u8"`

* (mostly) fully reproducible OS and user environments

* patching and/or locally building dependencies without ending up with a messy system

I can now finally get a (almost) 100% reproducible user environment including hardware configuration (kernel modules, disks, ...), installed packages, configurations, ...

home-manager [1] even enables declarative configuration of things like VS Code/Vim/Firefox plugins, and provides statically typed configuration options for many applications. The package repo is also very comprehensive.

For manually managed servers that aren't just bootstrapped from VM/cloud images the benefits are also wonderful.

It's definitely not all smooth sailing though:

* There is documentation, but it's spread across a lot of pages, disorganized and sometimes quite outdated or incomplete. I often have to read the source code - which is also mostly undocumented.

* Running unpackaged software: you can't just download binary releases and run them, unless they are fully statically linked (like most Go binaries). Instead you either have to whip up a package definition, or manually try to get the binary working (eg with patchelf)

* The language is somewhat quirky. As often the case with such niche languages, documentation and tooling are very subpar and debugging is not exactly a joy. I wish Nix was built on a more common language with good tooling around it. I bet it would be quite a bit more popular, due to the lower barrier of entry. You get used to it though if you are willing to put in some effort.

* Package ecosystem: the package repo is really comprehensive, but maintainership for some less popular ones is not up to Arch + AUR levels. Also the auditing for a lot of the packages is very limited. For any security critical environments or server deployments I would definitely limit myself to a hand full of core packages and/or be prepared for manual auditing. I wish the repo was split up more like Arch into core and community, with different guarantees for auditing and maintenance. (the Flakes effort will hopefully help here quite a bit)

All in all Nix is great though, and it's impressive how stable and featureful the ecosystem has become, despite being somewhat of a niche "product".

[1] https://github.com/nix-community/home-manager


nix work on windows natively yet?

No, bane of my life. It’s glitchy even with WSL2.

What problems have you had in WSL2?

What specifically? And are we talking, Nix or NixOS in WSL2?

Nix



Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: