I don't know much about Nix, although it has been on my radar for some time. The claim that the language performance is bad pops up time and again, and I'm sure many people would be happy to have a faster alternative --- so props to them for trying to address this clearly important problem.
That being said, in perusing related issues on GitHub such as [1], I have found no serious attempts to even profile the damn thing. The typical attempt seems to be throwing a random idea at the evaluator and crossing fingers. The example of a "performance experiment" [2] from the article is self-described as follows:
> Add an alternative impl of the now-abstract Bindings base class that is backed by a std::vector, somewhat similar but stylistically a little superior to the array-backed implementation in upstream nix.
I fail to see how that is a performance experiment when it does not mention performance at all.
The post also mentions issues with the general Nix design and a desire to modernize it, which I am unqualified to comment on, but seems a worthwhile endeavor in and of itself. I do hope that they won't end up writing a new evaluator for the Nix language without a proper understanding of what makes the current one slow and a clear plan not to repeat its mistakes, or the new one risks being just as slow as the old one.
The background on our side is that we did do quite a lot of profiling and investigation of the current evaluator - last year, informally, over video chats and IRC. We have a bunch of logs and various perf measurements spread out over some machines but right now the knowledge lives in our heads, mostly because at the time we did not expect this to spawn a serious project.
Once we get further along with the evaluator we will be incorporating some of that material into more specific blog posts, but it seemed a little too deep for the initial announcement.
Regarding that CL linked from the post: It was part of laying the groundwork for trying to have the same Nix language type backed by different implementations; the reason for this is that Nix's attribute sets (similar to dictionaries or k/v maps in other languages) have a wide variety of use-cases with wildly conflicting performance characteristics.
For example, one very common use-case is the data structure representing the top-level of nixpkgs: An enormous attribute set with many thousands of keys that is frequently modified & copied. An even more common one is related to the builtin function `listToAttrs` which deals with two-key attribute sets (name, value pairs). In current Nix, these are some of the biggest evaluator hotspots (apart from higher-level issues like cache-unfriendliness due to the tree-walking evaluation style) and optimising for one case vastly diminishes performance of the other.
The problem we ran into with all experiments of this form is that there is no clear interface separation in `libexpr` (the part of the Nix codebase that implements the evaluator): Everything is modifying and reading internal state of everything else, including in other parts of Nix that depend on libexpr. It made it basically impossible to really try anything without refactoring huge amounts of (memory-unsafe) code.
Now it's important to mention that the reason for this is understandable: Nix is an old project (18 years now I believe?) and I don't think anyone expected it to get to where it is now, so obviously different decisions were made in its implementation than would be made if it were started today knowing everything we know.
> The background on our side is that we did do quite a lot of profiling and investigation of the current evaluator - last year, informally, over video chats and IRC. We have a bunch of logs and various perf measurements spread out over some machines but right now the knowledge lives in our heads, mostly because at the time we did not expect this to spawn a serious project.
It's good to hear that you did do some profiling, even if the results are not public! Of course I would not expect the details in the initial announcement, but I would have expected a mention of this work. At least I have a much higher confidence in the success of this project now than just after reading the announcement.
Switching between the proper implementation of high-level concepts is a staple of untyped (and sometimes typed) runtimes, so in this context the changeset makes more sense than simply "replacing an array-backed implementation with an std::vector-backed implementation" as described, thanks for the explanation!
I'm also not putting any blame on Nix --- the fact that the performance of the evaluator is such an issue is in a way a testament to its success. Unfortunately large, long-lived systems do require active care to enforce proper boundaries between subsystems, and it's a shame if it has got to a point where it's no longer possible to reinstate that separation.
That being said, in perusing related issues on GitHub such as [1], I have found no serious attempts to even profile the damn thing. The typical attempt seems to be throwing a random idea at the evaluator and crossing fingers. The example of a "performance experiment" [2] from the article is self-described as follows:
> Add an alternative impl of the now-abstract Bindings base class that is backed by a std::vector, somewhat similar but stylistically a little superior to the array-backed implementation in upstream nix.
I fail to see how that is a performance experiment when it does not mention performance at all.
The post also mentions issues with the general Nix design and a desire to modernize it, which I am unqualified to comment on, but seems a worthwhile endeavor in and of itself. I do hope that they won't end up writing a new evaluator for the Nix language without a proper understanding of what makes the current one slow and a clear plan not to repeat its mistakes, or the new one risks being just as slow as the old one.
[1] : https://github.com/NixOS/nix/issues/2652
[2] : https://cl.tvl.fyi/c/depot/+/1123/