I wonder if one could write a truly statically typed tensor implementation with something like this. Where you can then say Tensor<f64, 16, 16, 2000> but also Tensor<bool, 32> and Tensor<f32, 4, 4> etc. Then some way of specializing for certain sizes to better use SIMD instructions etc. Even if it's just via something like `const if` inside of a single method, but with the assurance that the if is applied at compile time and thus e.g. illegal element access in the other branch is ignored etc.
Since the early 90's but in C++ through TMP and class template specializations. If you don't have many specializations, and those are sharing most of the code, you can simplify writing this code through if-constexpr since C++17.
But soon after you will realize that you need to do some operations with those tensors such as A + B + C. To compute this expression compiler will first need to compute (A + B) sub-expression and store the result into the temporary tensor. Only then it will be able to use this temporary result to compute the final expression by adding the C tensor. When you have a lot of such expressions to compute, and expressions are arbitrarily complex, you will want to avoid inefficiencies caused by creating many temporaries - there's an attached cost to them both in memory and in compute. And sometimes you will be able to fuse multiple operations into a single one.
This is a problem that can be solved with lazy evaluation and in C++ this can be done in compile-time through expression templates. And this is what basically all linear algebra C++ libraries are famous of doing.
In Rust you can implement Add for all kinds of combinations of types. I.e. when one of the parameters moves it's ownership you can just perform the operation in-place in that parameter and return it. So you can eliminate temporaries at compile time. It means much more work in the library, but the library user can just write A + B + C.
So Rust has that and C++ has template specialization, const if, and variadic templates. Would be cool to have all in one language.
> So you can eliminate temporaries at compile time.
> ... but the library user can just write A + B + C ...
> So Rust has that.
Eliminating temporaries in C++ is also done in compile-time through expression templates or more generally through TMP.
I'm pretty sure Rust type system is incapable to express many (?) of the TMP things from C++ - it does not even support partial specializations to begin with so it's hard to imagine that you can do a lot of TMP gymnastics. I think that this is an area where it is generally accepted that Rust is lacking against C++.
Yeah, I was listing a lot of things that C++ can express that Rust can't. But C++ doesn't have lifetimes so you can't express those. And you can use that to eliminate temporaries. But if you say that you can also somehow do that with template meta programming then great!
In C++, Eigen comes pretty close to this interface for defining vectors and matrices and mixing size-at-compile-time and dynamically sized quantities. And then, given the size of the vectors/matrices being used, dispatching the appropriate SIMD implementation.
You can do all except for the specialization in Rust for matrix + vector. You can do more in Rust about re-using temporaries (when one parameter to Add/Mul/... is a move). But you can't do arbitrary dimensional tensors or specialization for certain sizes in Rust. Also: Didn't know Eigen has tensor support now. I'm not actually very into this stuff, just recently played around what is possible in Rust in that context (and ran into compiler bugs in nightly).
That high-level idea is viable in many languages. I particularly like Zig's comptime as a clean way to implement it.
You often don't want that sort of specialization. Code like deep neural nets is a pain to handle (unless each layer is identical, which is admittedly moderately common) even with variadic generics and especially without, and it's hard for the compiler to generate the right specializations for (and the right number of specializations; anecdotally, you can expect enough code bloat that it causes problems).
Using branded types you can get a weak form of dependent typing and use that to get features like Tensor<f64, Dim<16>, Dim<MultipleOf<8>>, Dim<SensorChannel>>, where you're allowed to pass in concrete numbers, comptime-known constraints on runtime-known values, or even just weak information like that there's a particular brand (SensorChannel) on the type which allows you to match fully runtime-known values at compile-time and let the type-checker handle it (think of functions like matmul(Mat(A,B), Mat(B,C)), where the B has to be the same for both, but you very well might want to know it at runtime). It's still the library writer's job, unfortunately, to use `inline` and other such keywords appropriately to guide the compiler to the right number of specializations, but the API presented to people using that tensor library can be reasonably nice.
Incidentally, Zig _also_ makes that second thing (branded types) easy to write. I hacked a demo together [0] if you'd like to play around with the concept, but a dedicated tensor library should eventually offer something less verbose IMO.
If you're doing this with ML, my preferred strategy is to go the other way. Only write specializations that are easy for the chip to execute (e.g., appropriately aligned and length-aligned large powers of 2), and build code on top of that. If the computational kernel has to do padding or shifting or other garbage, even at the head/tail of a vector, then something in your model is horribly wrong anyway; just make that unrepresentable.
However what you can't do without dependent types (as far as I know) is infer or verify statically that the transpose of a Tensor<f64,20,10> is a Tensor<f64,10,20>.
Focus on the compile error story. In particular, the design should avoid anything that leads to post-monomorphization errors.
This is the single biggest mistake of Rust imho. I don't care in which compiler pass an error occurs as long as it happens at compile time.
Rust has this anyways, you can already create types that fail after monomorphization, so you don't get any of the touted advantages anyways. You might as well bite the bullet accept that the people who say “You guys are dumb, just copy Zig’s type system” have a point, and shed some of the ivory tower dogma in exchange for a better language. It's completely fine for monomorphization to fail, and it allows you to create much more powerfull types.
Rust is already incredibly complex, and most of that complexity comes from dozens of different systems that share few concepts all with their own syntax (modules, macros, traits, e.t.c.), so adding yet another syntax for static loops begs the question if some of this can't be unified.
Zig is a seriously flawed language, but comptime is the right abstraction for metaprogramming in an imperative language.
> You might as well bite the bullet accept that the people who say “You guys are dumb, just copy Zig’s type system” have a point, and shed some of the ivory tower dogma in exchange for a better language.
The fact that you call it "dogma" (and your other comments about Zig) suggests that you aren't going to be very open to changing your mind on this.
But as far as "the touted advantages" go, I've experienced them firsthand.
Being able to write generic code and know that if the code compiles in your library, it will compile for people who depend on it, is a pretty nice workflow. Being able to change that generic code and still be confident it will still compile for downstream users who download the new version is pretty nice too.
Being able to download some random dependency full of generic code, and to instantiate that code without having to dread reading mile-long call stacks that end with "foo.bar is not implemented in this function you didn't write" is pretty nice too.
(And yes, post-monomorphization errors can still happen in theory; I've never seen one happen in practice, certainly not with the frequency I saw the mile-long call stacks in C++.)
Those aren't some aesthetic details that language designers cling to out of some perfectionist desire to keep the language theoretically pure. It's part of what makes pulling dependencies especially painless in Rust.
There a million ways your code can break downstream dependencies, generic code is just one of them. Breaking stuff when you change things is just in the nature of it, which is why not changing APIs is so important. But I'd rather have the compiler errors you describe as something my IDE can jump to, than to spend hours poking around in opaque macro black boxes, because half of every library is a huge chunk of copy pasted code and macro magic to get around the type system.
Rust as is, is a de Bruijn criteria fulfilling proof language, with the macro system ending up as the meta language.
I don't want to discount the usefulness of definition-time type checking of generics, but (compile time) tests against archetypes are an obvious way to make sure that your generic code instantiates correctly.
Currently the compiler can choose to omit monomorphization, if the code is unused. If monomorphization can fail, the compiler needs to compute some unused monomorphizations, just to ensure they don't raise an error. This could hurt compile times. Otherwise you get compilation failures that depend on the exact way the compiler decides which monomorphizations are used. Plus otherwise semver compatible changes in a dependency can surface latent monomorphization failures, breaking the build.
However I do overall agree with you, that rust needs to accept post-monomorphization errors. I see no way to get usable const-generics otherwise.
Yeah, It's not that I don't see the advantages. I wish that there was a way to get our cake and eat it too. Maybe we can dependently typed eat our way through to the other end, where while we do have monomorphization errors, having constraints on generic types feel natural and ergonomic, much more like in a system like Lean.
I think what it boils down to at the end is where you want your pain to be, pains from constantly hitting limitations and providing an escape hatch like macros, or pains from having a system that is too powerful and all the chaos that brings.
I'd rather reign in the chaos, after all we also choose to be Turing-complete, and not always terminating like Coq, but mmv.
Priorities. The team is too focused on global maxima, speed and reinventing compiler design at the cost of a reference implementation that's fundamentally broken, and too complex to fix.
This results in miscompilations, broken buildins, features added that are then ripped out again.
I think fast iterations and experimental stuff are ok if you have a stable core.
I.e. I don't mind if zig decides to change its for loop syntax after 5 years, so long as there is a simpler stable construct like `loop { break; }` that people that use the language in production and library maintainers can use.
You need a certain critical mass for a language to be self sustaining, and that requires people to actually use your stuff, which in turn requires at least some stability guarantees for some subset of the language.
Zig's response to that is "I'm just my inventors toy yet, but it will be grand when version 1.0 hits in 10-50 years."
The fact that they now want to replace LLVM with their own backend makes me think that even 50 years would be optimistic.
Go did not have the goal of the best possible performance, it was targeting good performance with amazing compile times. It still targets this. It was also developed by some of the most productive geniuses in the world (disclaimer: the go team used to report up through me)
Getting to 70 percent of llvm will be relatively easy for zig and like all others, they will congratulate themselves and pretend it validates their choice. They will then discover the other 30 percent happens 0.1% at a time, because there are no silver bullets. This will take them 10+ years because it always does.
Yeah, LLVM is here to stay for Zig. The plan is to have a custom backend for fast Debug builds for use during development, while emitting LLVM bitcode for release builds that can then be fed to LLVM.
Not linking to LLVM would also make the barrier to entry on contributing to Zig itself lower, because now you can just compile Zig with Zig, and not worry about other dependencies.
All of this would've come for "free" (lets and lots of developer hours) if introspection is further developed. It's truly a tragedy that things went down how they did.
Variadics are a prerequisite for this sort of introspection, not an alternative. The design you linked uses an ad-hoc keyword to hack around their lack.
I agree, but I think there's a possibility of keeping the variadicity only as introspection "compiler magic" and coming up with a higher level implementation of variadics for the general language that drops naturally out of introspection. My hope is that eventually variadicity will just be "only library code" that you can just inspect how it is implemented via introspection.
I cant think of a single real use case for variadic generics.
When you're talking about functions operating on mixed, variable length, types, what meaningful logic can you actually apply, that doesn't fall into requiring the types to implement an interface anyway?
When you're talking about struct/classes/etc, what do you need over and above defining a class/record/dataclass? It would seem to me, you're just skipping the part where you give your tuple a name, but is that even a good idea to do?
To be clear, I did not find the examples in the post convincing, as they represent mostly wrappers/adapters for implementing or consuming arbitrary types, but there are no examples for how that might actually be useful downstream.
I can think of plenty real world usecases. Take for example a network interface that can send or receive predefined packet structs.
Imagine each packet requires a size header. When you want to send multiple packets at once, you now want to optimize that and only write a single initial header to the interface, preceding the data.
With variadic generics, you can enable a syntax like `interface.write_packets(packet1, packet2, packet3, /* ... */);` which writes the packets in the desired optimized way. It can internally construct a serializable data tuple from the variadic generics and add the correct header only to the beginning.
Without variadic generics a similar syntax is only possible with macros, which means that it cannot be implemented as a reusable trait.
> It would seem to me, you're just skipping the part where you give your tuple a name, but is that even a good idea to do?
Sometimes you explicitly don't want to require naming the tuple for flexibility, like in the example above.
I think the utility is strongly dependent on the kind of code you write and the sophistication of the metaprogramming your code will benefit from. Many C++ programmers made similar comments when it was introduced with C++11 and a few still maintain it has no real use case. However, for C++ library writers it was a game changer. In practice, they replace a lot of use cases for code generators. It took me a while to get into variadic generics, but now that I understand them I would feel crippled using a language without them.
A lot of the value is to derive complex structs programmatically that may never be realized concretely but feed other metaprogramming algorithms that slice and dice them and make decisions based on the properties of the individual types. You don’t want to name them because (1) there may be hundreds from an unbounded set and (2) your may never need to reference them directly, so it saves a mountain of code and eliminates a massive amount of edge cases. At their best, you can use them to express insanely complex things with surprisingly little code.
It is not the kind of feature you use profligately in a code base. But when you need them, you really need them. Most of what they can express in ordinary code would otherwise require complex code generators, compiler magic, or outrageous amounts of boilerplate.
I'm writing a library at the moment with exactly this kind of issue (not quite that formulation, but morally equivalent). The whole thing would be a lot simpler (and remove the need for procedural macros to generate all of the types and impls) with variadic generics.
Would I use variadic generics regularly? Probably not. But in places where they'd be useful - boy, would they really be useful!
because you can make that a single macro which kinda is a bit like variadics. Through it still is limited to some specific number of your size (typical 16 or 32).
Anyway not having varadic generics by itself is in most cases just a small inconvenience with limited benefits and there are many far more important priorities IMHO.
The main benefit for varadic generics is that they might unblock some other improvements on top of them.
> I cant think of a single real use case for variadic generics.
Even if you could think of one, I personally think the rust community does a poor job of weighing the tradeoff between supporting a niche feature and adding complexity to the language. Rust is already so huge that all but the most dedicated have little hope of understanding even half of the language features. I'm not sure I understand the incentive structure that makes it this way, but it is hard for me to see how rust avoids becoming the next c++ unless they learn how to say no. Honestly I believe it's already too late and that rust's best ideas will be adopted by a smaller language with better ergonomics, compiler performance and learnability that will eat its lunch.
The status quo is that people who would like to use variadics but can't, instead settle for complicated ad-hoc macros. Those are far worse for "ergonomics, compiler performance and learnability."
- it is duck typed, which to be fair macros are also, but variadic generics would be type checked at the definition;
- it interacts badly with traits, that is if you allow both querying for implemented traits and creating new trait `impl`s then this can create cycles in the execution of comptime blocks, meaning the compiler needs to compute a fixpoint or arbitrarily give an error.
Also, Rust has lifetimes which are erased prior to monomorphisation—so specializing based on them is unsound. Any `comptime` equivalent in Rust would have to limit the access of `comptime` blocks to lifetime information in some way.
> When you're talking about functions operating on mixed, variable length, types, what meaningful logic can you actually apply, that doesn't fall into requiring the types to implement an interface anyway?
Anything higher-order. At the most basic level you want to be able to take a bundle of values and a bundle of functions and apply the functions to the values, or take a bundle of functions and another bundle of functions and make a bundle of composed functions. You will probably complain that these examples are trivial, but if we don't have the syntax or vocabulary to handle the trivial cases then how can we even start to think about the more complex cases?
> When you're talking about struct/classes/etc, what do you need over and above defining a class/record/dataclass? It would seem to me, you're just skipping the part where you give your tuple a name, but is that even a good idea to do?
It is, if only because you want to be able to transform existing records generically rather than having to concretely write out every specialised possible version of your record. For example, think of writing a generic "patch" method for use in the "update" route of a CRUD webservice - something that lets you call it with an existing record (e.g. a user) and some representation of a few deeply nested fields of that record (e.g. the postal code that's on that user's address) and get an updated version of the record, without having to write out the type for that "patch" input longhand. This is a basic, trivial thing that people do in TypeScript every day, but you still can't even write a type for it in Rust, yet alone implement it.
> I think variadics aren’t necessarily useful for application developers, or people working on libraries with mostly concrete types (eg web libraries). But for people working on libraries with lots of type manipulation, variadic generics have obvious benefits. They let you go from manipulating types to collections of types.
One example is in SQL libraries. If you have a trait `DatabaseColumnStorable` for types you can store in a database column (integers, strings, etc), then with variadics you could have a `struct DatabaseRow<...Columns: DatabaseColumnStorable>(...Columns)` to represent rows of a database table in a type-safe manner.
For a quick example, Bevy needs to implement some trait for functions with a variable number of inputs. However it doesn't stop there, it also needs to store some state for each input, in other words it need to map that `fn(...T)` into a `(...State<T>)`. How do you model this with generics? When I tried to think about this every way I found to express this didn't allow the consumer code to call any methods on those `State`s, essentially making this useless.
In https://GitHub.com/celtera/avendish it's used to do operations on (heterogeneous) I/O ports of data flow objects (simply represented by variables of basic types.. float, int, string, etc.). This library would be entirely impossible in c++ without that feature.
With variadics the types would still implement an interface, they just allow you to work with those types t compile time rather than type-erasing them into a dynamic trait object.
