FWIW my take is not that Yuri is expressing "there are too many bugs" so much as he's expressing a problem in the culture surrounding Julia itself:
> But systemic problems like this can rarely be solved from the bottom up, and my sense is that the project leadership does not agree that there is a serious correctness problem.
Concisely:
1. The ecosystem is poorly put together. (It's been produced by academics rather than professional software developers.)
2. The language provides few tools to guarantee correctness. (No static typing; no interfaces.)
Personally, what I'd love to see is one of the big tech companies come on board and just write their own ecosystem. The Julia language is amazing. The ecosystem needs to be rewritten.
Lots of things are being rewritten. Remember we just released a new neural network library the other day, SimpleChains.jl, and showed that it gave about a 10x speed improvement on modern CPUs with multithreading enabled vs Jax Equinox (and 22x when AVX-512 is enabled) for smaller neural network and matrix-vector types of cases (https://julialang.org/blog/2022/04/simple-chains/). Then there's Lux.jl fixing some major issues of Flux.jl (https://github.com/avik-pal/Lux.jl). Pretty much everything is switching to Enzyme which improves performance quite a bit over Zygote and allows for full mutation support (https://github.com/EnzymeAD/Enzyme.jl). So an entire machine learning stack is already seeing parts release.
Right now we're in a bit of an uncomfortable spot where we have to use Zygote for a few things and then Enzyme for everything else, but the custom rules system is rather close and that's the piece that's needed to make the full transition.
The fact that things are being rewritten and the primary criteria being looked at is speed IS culturally a big part of the problem. If you don't prioritize provable correctness first, then I guarantee that the code is not correct. And as the complaint explains, incorrect code costs people months and leads them to not trust the result.
Don't believe me? Re-read the blog post about how a major source of bugs is people making assumptions into silent errors by removing bounds checks. Simply being able to re-run the same code in a slow mode with the bounds checks turned back on would undoubtably catch bugs.
100% this. In a discussion on "cultural correctness issues prevents me from using Julia", it's very telling that the response is "more speed!"
There's been a decent number of posts based around "Julia has these problems". And I don't think that's because the world at large has a vendetta; I think it's because the world at large desperately wants to use Julia, but struggle with hard blocks that are currently preventing adoption.
FWIW I do think there's a growing acceptance in the Julia community that these concerns are real, which is good. (See the parallel discussion on the Julia Discourse.)
Two of the mentioned packages, Lux and Enzyme, have increased correctness and decreased API surface... and were not mentioned for speed (though a lot of things end up faster when it's easier to prove correctness in the compiler)... so the response wasn't "more speed" but "here's correctness with resulting speed"...
Actually Enzyme was mentioned for speed, not correctness. To verify, go back and see that you wrote, Pretty much everything is switching to Enzyme which improves performance quite a bit over Zygote...
You didn't mention speed on Lux, but it is a rewrite. The rule is that a rewrite should be assumed to be buggy until proven otherwise. A culture of having everything permanently under construction comes with upsides and downsides. And unless you have good testing, correctness problems is one of the downsides.
> Re-read the blog post about how a major source of bugs is people making assumptions into silent errors by removing bounds checks. Simply being able to re-run the same code in a slow mode with the bounds checks turned back on would undoubtably catch bugs.
Running Julia with the command line argument --check-bounds=yes does that, and package testing always uses this option to disable inbounds.
In most cases, being fast is useless without being correct. Even approximate things like simulations depend on the programming language being deterministic and correct. Otherwise the math of approximation doesn't work out.
With my programmer hat, the first thing I care is not speed for most cases. Unless there's an explicit need for speed, I don't select the language I gonna use with respect to its performance, and I don't port a tool unless the speed becomes limiting.
It's important to make it run first, then make it fast. Otherwise, things go very wrong, very fast (pun intended).
But that feature already exists: you can re-run the same code in a slow mode with the bounds checks turned on... It is just a flag you can set at startup.
Enzyme dev here, so take everything I say as being a bit biased:
While, by design Enzyme is able to run very fast by operating within the compiler (see https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b... for details) -- it aggressively prioritizes correctness. Of course that doesn't mean that there aren't bugs (we're only human and its a large codebase [https://github.com/EnzymeAD/Enzyme], especially if you're trying out newly-added features).
Notably, this is where the current rough edges for Julia users are -- Enzyme will throw an error saying it couldn't prove correctness, rather than running (there is a flag for "making a best guess, but that's off by default"). The exception to this is garbage collection, for which you can either run a static analysis, or stick to the "officially supported" subset of Julia that Enzyme specifies.
Incidentally, this is also where being a cross-language tool is really nice -- namely we can see edge cases/bug reports from any LLVM-based language (C/C++, Fortran, Swift, Rust, Python, Julia, etc). So far the biggest code we've handled (and verified correctness for) was O(1million) lines of LLVM from some C++ template hell.
I will also add that while I absolutely love (and will do everything I can to support) Enzyme being used throughout arbitrary Julia code: in addition to exposing a nice user-facing interface for custom rules in the Enzyme Julia bindings like Chris mentioned, some Julia-specific features (such as full garbage collection support) also need handling in Enzyme.jl, before Enzyme can be considered an "all Julia AD" framework. We are of course working on all of these things (and the more the merrier), but there's only a finite amount of time in the day. [^]
[^] Incidentally, this is in contrast to say C++/Fortran/Swift/etc, where Enzyme has much closer to whole-language coverage than Julia -- this isn't anything against GC/Julia/etc, but we just have things on our todo list.
[ps sorry if this ended up as a dup, I meant to reply deeper in the tree, so I deleted the older comment and moved it here].
With luck you will succeed. And that is a great thing.
But I maintain my position. If users are choosing packages because of speed without worrying about correctness, then packages will become popular that care less about correctness than what you describe. And when people combine popular packages that make conflicting assumptions, correctness will be lost.
In other words the problem is the attitude, not the specific package. For another example of the same problem, look at how C/C++ compilers prioritizing speed has resulted in their taking advantage of undefined behavior in a way that makes it far harder for any significant C/C++ codebase to be correct.
> The Julia language is amazing. The ecosystem needs to be rewritten.
I think this is pretty unfair. Julia has many libraries that have allowed me to build things that would have taken orders of magnitude more effort to produce in other languages with the same conciseness and efficiency.
Composability and efficiency hard. Are things better elsewhere? Python has excellent libraries. But these are big monoliths that not only do not compose well, but are also hard to understand deeply as they are essentially a thin layer over C, C++, Fortran, etc.
Julia simply needs more maintenance and more tests. There is no big corporate backing, and things depend on individual efforts. In my opinion, most packages are already polished and easy to understand.
IMHO, the biggest problem is that there is no reliable library to build huge transformers.
The answer as far as I can see is to write integration tests when you want to use composability (and/or just check that the packages in question already have integration tests against each other -- increasingly many do). It's not especially hard or anything, but you need to know to do it.
Hm. Some cases are where non-composable things might just fail; that is indeed "not especially hard or anything" to detect with simple integration tests, sure. But those you'll also probably notice without integration tests too, when your code just obviously doesn't work. Either way, sure, you'll find out about this quick, and know you have to fix it if you want to use that code to get the results you wanted to get.
But some of the examples in OP of non-composability are where edge cases (not all cases) will give you the wrong answers. In the worst case, in ways that aren't immediately obvious.
I think it's too optimistic to say that you just need to know that you should write some integration tests and then write some easy and obvious ("not especially hard") ones, and then you'll always catch these easily and early.
Sure, but that's a caveat that applies to testing in general, not something somehow special to composability. Bugs that only appear in edge cases or give wrong answers are always a concern when writing tests, that's nothing new.
I don't actually even see any evidence here that these sorts of bugs are more common or more insidious than any other sort of bug, rather it looks to me that since dispatch-based composability is a bit of a new phenomenon, people haven't always been on the lookout for these bugs yet, or learned the patterns of where and what to test for to find them -- but once you've seen a few cases, patterns start appearing and it becomes more clear what you need to test, just like anything else.
The broader issue to me is that I think people often underestimate the degree to which Julia is a fundamentally different programming paradigm than any other language most people have used before (unless you've spent a lot of time in maybe CLOS or Dylan) -- there's not even an agreed-upon name for it, but I'd call it "dispatch-oriented programming". Often times people will come in to Julia expecting all their expertise and intuition from class-based OO or whatever other paradigm to just effortlessly transfer to Julia, and that's really not the case, which tends to lead to frustration.
Right, it's a caveat to testing in general that shows "well, just write integration tests, it's not especially hard to do" is not a satisfactory solution to the problems discussed in OP with specific actually occuring examples. You were the one suggesting it was, not me!
Well I don't think it's hard any more than writing any other sort of test in a given language. That doesn't mean it doesn't require expertise in that language.
I used to love javascript. When people asked, I made a similar argument - that you need tests anyway, and if you're writing tests you'll spot any obvious typing bugs in your javascript code.
I think I was wrong. Moving to typescript has been a revelation for me, because I find I need far fewer tests to make working code. (And my code so often just works as soon as it compiles!). I can code much more fearlessly.
Rust is similar. By the time the borrow checker has sufficiently bruised my ego, my rust code often just works. And thats a joy.
What I'm hearing is that Julia is more like javascript and C, and less like typescript and rust. That in Julia you need to be more paranoid around correctness - because you can easily compile a binary that produces incorrect results. You need extensive integration tests to guard against that, because its not enough to know that two libraries work individually and compile together. They need to also be tested together or they might silently do the wrong thing.
That sounds like a bad experience. Honestly, that sounds worse than javascript. At least in javascript, type errors almost always result in instant crashes with a useful stack trace. You say testing Julia code isn't hard, but that doesn't mean I want to do it. Building a stone wall by hand isn't hard either, but I'll take my desk job every time.
Just to make sure we're on the same page though, it's perhaps worth clarifying that this particular integration issue isn't something you have to worry about every time you use two packages together (far from it), it's only in the case where you're trying to use a custom type from one package (e.g., one which replaces base Arrays or base Numbers or such) in functions from another package.
> Are things better elsewhere? Python has excellent libraries. But these are big monoliths that not only do not compose well, but are also hard to understand deeply as they are essentially a thin layer over C, C++, Fortran, etc.
I dunno.
Things like the use of scipy.spatial.distance metrics[1] by in sklearn clustering[2] seems a great example of composability that is easy to learn and very efficient.
And the sklearrn side isn't a "thing layer over C, C++, Fortran" even if scikit is (sort of) this.
How has Python - almost surely the most successful and widely adopted scientific programming ecosystem - avoided the problems of #2? E.g. Python doesn't have static typing.
Is it just that Python is so widely used there's institutional support for incredible linting and type check tools despite the lack of static typing? Or that much of the science/data ecosystem of Python is written in lower level statically typed languages?
(sadly possibly necessary edit/clarification: I'm not trying to be That Guy who answers every complaint about Julia with a matching complaint about Python. I'm legitimately curious about how Python got where it is without static typing, and what that implies about paths to a better ecosystem for Julia.)
Julia heavily makes use of multiple dispatch among with other convenient type related features much more complex than Python, to a point where they are often abused and sometimes have uncaught edge cases. It makes the language very powerful but has its downsides.
And to be fair to Python, static analysis has come a very long way and the CPython interpreter makes far fewer complex assumptions than the Julia compiler. It’s also fairly strongly typed as well, so I’ve found that challenges with the type system cause more issues with packaging and maintenance than it does correctness.
Indeed Python is relatively simple language. It also adheres to the principle of least astonishment and dynamic types are used towards this goal. Finally it does not mind making large changes to the language.
With Python most of the time when you have an unexpected result with the language or a library it is often a matter of realizing "OK that's the way it works", and moving on with your work. The language and libraries strive so much to always return sensible results that they are fewer instances when you would call a behavior a bug.
I don't think it needs a rewrite as much as careful maintenance from people who have time to dedicate to software quality. Most of the APIs are good, it's just that a lot of the code is under-tested and doesn't receive enough love. Having more big companies using Julia would help a lot with that.
> But systemic problems like this can rarely be solved from the bottom up, and my sense is that the project leadership does not agree that there is a serious correctness problem.
Concisely:
1. The ecosystem is poorly put together. (It's been produced by academics rather than professional software developers.)
2. The language provides few tools to guarantee correctness. (No static typing; no interfaces.)
Personally, what I'd love to see is one of the big tech companies come on board and just write their own ecosystem. The Julia language is amazing. The ecosystem needs to be rewritten.