
Flakes – Proposed mechanism to package Nix expressions into composable entities - based2
https://github.com/NixOS/rfcs/pull/49
======
cwp
This is really interesting. Right now you can't really compose nix expressions
from several different sources. There are ways of doing it, but they all
involve one hack or another that limits the usefulness of the result. This is
why nixpkgs is a huge monolithic repository.

Flakes would allow package definitions from different sources play well with
each other. This means a bunch of things:

1) Nixpkgs could be broken up into several package sets for better
scalability. Eg, haskelPackages and pythonPackages could be broken out into
separate repositories.

2) Package authors could provide package definitions along with the source.

3) Companies could have a flake for internal packages without having to patch
or overlay nixpkgs.

4) Alternatives to nixpkgs can get started without having to supply _all_ the
packages that a user might need.

5) You could even create libraries of nix functions that supply no packages at
all, but get used by other flakes to define packages. For example, the
machinery for composing a python environment.

~~~
ris
> 1) Nixpkgs could be broken up into several package sets for better
> scalability. Eg, haskelPackages and pythonPackages could be broken out into
> separate repositories.

While this _could_ happen, it's exactly what I hope _won 't_ happen, because
in my opinion the idea that haskell or python packages are fundamentally
different and can live in their own world is a fallacy. I spend quite a bit of
my time fixing python packages on nixpkgs master and the number of times the
fix actually involves making a tweak to a non-python library means I would
forever have to be playing games synchronizing repositories. Also, gone would
be the ability to make atomic changes across these boundaries.

Viva la monorepo.

~~~
Apaec
> Viva la monorepo.

Absolutely. Look at edolstra take on this though
[https://github.com/NixOS/rfcs/pull/49#discussion_r305450668](https://github.com/NixOS/rfcs/pull/49#discussion_r305450668)

This is why I'm scared of this change.

~~~
spindle
Me too. I think (hope) that the NixOS community won't end up doing all the
work it would take to break up the existing monorepo; instead they'll just add
a few small additional repos.

------
tbenst
I’d love for a way to write nix expressions that easily lead to reproducible
builds. Of course this is possible right now next, but requires substantial
legwork to pin Nixpkgs version or add SHA for every tarball. Especially now as
the machine learning story for NixOS is becoming strong, it would be awesome
to have fully reproducible model training using such a system. This highlights
a difference from docker or nvidia-docker, which do not control driver
version.

~~~
colemickens
Fortunately, `builtins.fetchTarball` makes this easier. I do this in my
`nixcfg` repo so that I can build my entire machine config (patches and
enabling Iris in Mesa, and all) on a stupidly-cheap Packet.net VM:
[https://github.com/colemickens/nixcfg/blob/master/default.ni...](https://github.com/colemickens/nixcfg/blob/master/default.nix).

If you follow the rabbit-hole: `default.nix` -> `lib.nix`, etc, you can see
that I pin nixpkgs, have an update script that updates the nixpkgs revs I
build against, it supports building against a custom, local `~/code/nixpkgs`
if it exists, and I have my machine config abstracted out to where I can build
a "GNOME instance of my machine" or by default, my machine with Sway and sway
related packages installed. Much of this should be easier with flakes, as I
understand it.

It's not the cleanest config, the README needs love, but maybe it can be
inspiration in the meantime. :) My latest trick was figuring out how to get
the new Mesa Intel Iris Gallium driver enabled without rebuilding the world,
and I extracted it to what I call a "mixin" that anyone can copy and just use:
[https://github.com/colemickens/nixcfg/blob/master/modules/mi...](https://github.com/colemickens/nixcfg/blob/master/modules/mixin-
intel-iris.nix)

(And technically I still use mozilla/nixpkgs-mozilla to pull the latest
Firefox Nightly which is impure and thus not always perfectly, perfectly
reproducable. I do however pin the overlays themselves also - like my nixpkgs-
wayland overlay that packages HEAD versions of Sway and other Wayland related
tools!)

~~~
tbenst
This is interesting. Sounds like you do this on the system level? I’m
interested in doing this on a shell or package level eg I’d love to have
system on bleeding edge but be able to jump back to same binary blobs
associated with a particular revision of software

~~~
colemickens
`nix-env` can be coerced into pinning a package at a specific revision,
effectively ignoring channel updates. However, yes, I do all of my nixos
package management in my system configuration and try to avoid `nix-env` at
all costs as it leads to drift. However, in at least one case, this is largely
un-avoidable (in the case of OBS Plugins, there's no NixOS infra to make sure
that 'wlrobs' is available at a well-known spot for OBS to pick-up. We'd need
to make a plugin-aware wrapper, or `programs.obs` module probably. This is
because nix-env does some additional symlinking that isn't done for system-
installed packages).

------
codedokode
While Nix's concepts are awesome, the implementation doesn't look so nice.
Their "language" for writing Nix files [1] doesn't look attractive. Doesn't
make me want to write a package description.

Of course, it is better than languages with lots of brackets, but still not
good enough.

[1]
[https://github.com/edolstra/nixpkgs/blob/release-19.03/flake...](https://github.com/edolstra/nixpkgs/blob/release-19.03/flake.nix)

~~~
vertex-four
For basic package descriptions, the language is not terribly different from
JSON. Here's a definition of a Python application I'm working on:

    
    
        { pkgs ? import <nixpkgs> {}}:
        
        with pkgs;
        
        python3Packages.buildPythonApplication rec {
          name = "quicktill";
        
          src = ./.;
        
          doCheck = false;
        
          propagatedBuildInputs = with python3Packages; [
            dateutil pyyaml
            sqlalchemy psycopg2
            pycups reportlab
            requests_oauthlib
            pygobject3
          ];
        
          nativeBuildInputs = [ wrapGAppsHook ];
          buildInputs = [ gobjectIntrospection gtk3 ];
        }
    

The only big things here are the fact that this is defined as a function that
takes the package repository as an argument to access dependencies (the first
line), and the "with" expressions that allow me to skip having to type "pkgs."
and "pkgs.python3Packages." for every package I depend on.

~~~
jforberg
That looks completely inscrutable to me. Like a sequence of magic words. I
don't see the connection with JSON at all.

~~~
vertex-four
I'm not really sure what you're finding inscrutable. I'll tear out the
boilerplate that explains that this file is a function and returns a Python
application (which is documented in the nixpkgs manual and can just be copy-
pasted if you really don't want to figure out what it does):

    
    
        rec {
          name = "quicktill";
        
          src = ./.;
        
          doCheck = false;
        
          propagatedBuildInputs = with python3Packages; [
            dateutil pyyaml
            sqlalchemy psycopg2
            pycups reportlab
            requests_oauthlib
            pygobject3
          ];
        
          nativeBuildInputs = [ wrapGAppsHook ];
          buildInputs = [ gobjectIntrospection gtk3 ];
        }
    

Is that any better? It looks an awful lot like some dialect of something JSON-
like to me - there's some different symbols used and array values are
separated with spaces rather than commas, but aside from that it's just a
record of key-value pairs.

From there, the first line in the original example says "the following
expression is a function that is called with this parameter", and the
parameter is an object containing a key called "pkgs", which is defaulted to
the result of "import <nixpkgs> {}". This is explained in more detail in a
handful of lines in the language manual.

Functions are called just by referencing the function and then the arguments
separated with spaces, like Ruby or Haskell - so
python3Packages.buildPythonApplication is a function that is called with the
object above.

Finally, "with foo;" just means "every key in foo can be accessed without
typing foo. in the next expression".

The result is basically: this file defines a function which takes an object
containing the package repository as the "pkgs" key, and returns the result of
calling pkgs.python3Packages.buildPythonApplication with the object defined in
the rest of the file.

It's a lot more derived from the syntax of functional languages than C-like
ones, but that's not necessarily a bad thing. Also: this is basically the
entire language. There's a couple of different string syntaxes and some stuff
for merging objects together, and also a mechanism for temporarily naming
things that aren't part of an object, but in general... there's not a lot more
to it. Most of the complexity is in the domain itself - what's the difference
between a propagated, native and regular build input, how do you manage
plugins and options for a package, etc.

What's your preferred packaging DSL?

~~~
jforberg
I mostly deal with openembedded at work and I have few positive things to say
about it. I'm sure Nix is much faster and nicer. Frankly it's hard to imagine
that the opposite could be true.

But your example looks nothing like Json.

~~~
vertex-four
I'm not sure what to say. Replace the = signs with colons, the semicolons with
commas, stick commas between array elements and stick everything between
quotes and you basically have json.

~~~
tannhaeuser
That looks like a Turing-complete DSL. Not like JSON at all. Do we need yet
another Turing-complete language for writing package metadata?

~~~
viraptor
The example lacks functions / flow-control statements (ok, apart from "import
if not defined", but it's more of a default value than a branch) / references
to other elements. It's pretty much key-value dictionary. Not sure how you got
an impression of Turing-complete DSL from that, since it would require at
least one of them.

~~~
weberc2
The Nix expression language has branches and functions (including recursive
functions). It is Turing complete, and it probably is necessary for its
charter.

~~~
viraptor
OP said it "looks like" a Turing complete DSL just from the example.
Regardless of what extra functionality the actually is, the examples didn't
suggest it.

~~~
weberc2
I guess I was giving the benefit of the doubt, since that's such an irrelevant
nit to pick...

------
kalium-xyz
Looks good, but makes me worry that target sponsored this. Are RFCs normally
cooperate sponsored?

~~~
eridius
Target is also behind lorri:
[https://github.com/target/lorri/](https://github.com/target/lorri/)

I view corporate interest as a good sign for the continued health of the
ecosystem. As long as they don't "take over" the project there's no harm. And
in particular, publishing an RFC like this shows that they're doing things the
right way and cooperating with the community.

------
Apaec
This proposal makes me scared for my bet on going full with NixOS.

To me it seems like a really complicated technical solution to a social
problem(the monorepo approach doesn't scale).

If more people are needed to maintain the nixpkgs repo why instead not focus
on ways to increase the community size.

Missing docs have been a problem for NixOS for a long time, most awesome
features are lost on some proficient users tweets. Having better docs would do
wonders for the community.

I think if NixOS wants to copy some of Rust strengths, quality docs should be
its first priority.

~~~
ris
> the monorepo approach doesn't scale

Citation needed.

~~~
Apaec
That's mentioned by edolstra in a comment on the proposal.

[https://github.com/NixOS/rfcs/pull/49#discussion_r305450668](https://github.com/NixOS/rfcs/pull/49#discussion_r305450668)

~~~
gmfawcett
I know who the author of that comment is, but it is not a universally held
opinion among Nix users.

------
kreetx
It will replace nix channels? Looks like a big change!

