So, looks like Julia would be an easier transition for a lot of academic scientists. What am I missing? I mostly use R and Python. Can anyone tell me briefly why I should use Julia over Python?
I just wrote my first bit of Julia this weekend. I'm impressed - it's clearly been designed for technical/numeric computing with modern language features baked-in. The type system and macros are some significantly distinguishing characteristics as compared to Python. The Unitful.jl package illustrates how these can be used to powerful effect. Multiple dispatch is another such characteristic. Having benefited from static type systems in other software projects, I would never again want to write significant technical code without a first-class type system supporting declarative type constraints.
If I could never write Matlab again and instead use Julia, I'd be delighted.
Well written pure julia programs tend to be about as fast as C, or in other words about 100 times faster than Python or R.
Before that gets you too excited or sceptical, let me say that end users will not really see many significant speed boosts. When I say end users, I mean people who are just loading packages and writing scripts where they plug variables into functions from the package. The reason these end users won't see much difference is that a well made Python or R package (including things like Numpy) are actually Python and R wrappers around C, C++, Fortran or Julia code. However, if you are trying to do something that your packages weren't directly designed to do and writing non-trivial code, then you're going to start seeing the speed advantages of Julia.
With all that in mind, I'd say the main reason for end users to use Julia is not the speed, it's the expressiveness and the top notch libraries.
Expressiveness is kinda a vague concept, but what I'm basically saying is that through things like multiple dispatch and macros, julia is able to provide ways of writing programs in almost any domain that feel incredibly natural and various programs will compose with eachother in ways you won't see in any language except Common Lisp.
Finally, because of Julia's native speed and expressiveness, it doesn't get in package developers way like Python and R do and so we're seeing that even though julia has a much smaller community than Python or R, we have a package ecosystem that's quite comparable and in certain specific regions, flat out superior. As adoption grows, we're going to see the julia package ecosystem accelerate and Julia will make sense for more end users.
For now, whether or not Julia makes sense for you really depends on what field you're working in (ie. if there are good packages already for the tasks you're interested in) and how advanced a programmer you are / want to be (ie. if you're going to be trying to do something that's not already covered by existing packages).
> However, if you are trying to do something that your packages weren't directly designed to do and writing non-trivial code, then you're going to start seeing the speed advantages of Julia.
Or you use Numba. :-)
But yeah, the speed can be pretty bad compare to native code if you can't compile everything to native with Numba. To put some concrete numbers here, the one time I had some complicated but heavily-optimized Python and C++ scicomp codebases computing exactly the same thing, I saw [1] running times like the following depending on the algorithm:
1. 10.6s with NumPy, 7.5s with NumPy + Numba, and 0.6s in direct C++ (1.4x and 12.5x respectively)
2. 68s with NumPy, 6.5s with NumPy + Numba, 0.5s in direct C++ (10x and 13x respectively)
The first algorithm was simpler and more vectorizable, while the second algorithm was more complicated and less vectorizable but with a lower time complexity. In either case, the algorithm still had to do a fair bit of work in Python, so it wasn't running native code 100% of the time (which I think is fairly realistic).
In either case, you can see the difference between C++ and Python was around 13x when using Numba... not quite as bad as 100x, but I imagine Julia probably does better.
[1] Actually, originally I saw only a 5x improvement with Python 2.7 on my older laptop, but I re-ran these on my current laptop under Python 3.7 and the difference is 13x now.
My gripe with Numba is that you're still giving up on a huge suite of Python's language features. One could say "oh well, eventually with enough work Numba will make all of Python as fast as Julia" but I don't think that's true. Python has specific semantics core to it's design that make all sorts of optimizations impossible. Julia made a lot of core design decisions to make sure that these optimizations were possible. You could invent a new language that's python-like but doesn't have the same limitations but then you might as well just use Julia.
Now, if all you ever interact with is is floating point numbers then Numba may be enough for you unless you're in an application like differential equation solving where the context switch of going back and forth between interpreted Python code and compiled Numba code will kill your performance.
But Julia's compiler will work on custom types just as well as builtin types (in fact, Julia's builtin number types are written in pure julia and could have been implemented in a package with the same performance).
> In either case, you can see the difference between C++ and Python was around 13x when using Numba... not quite as bad as 100x, but I imagine Julia probably does better.
When I said the 100x that's mostly a statement about Python's looping performance and discounting Numpy vectorization Ie. comparing a Python for loop to a C or Julia for loop (arguably a strawman, I know). Numpy is just calling C routines afterall, so for the most part it can be really fast other than the fact that individual Numpy calls don't know about each-other which precludes some optimizations.
But yeah, claims about performance differences between languages is a really technical and highly context contingent debate which is why I wish people wouldn't lean so much on Julia's speed as it's main selling point. It will mostly just lead to disappointment or confusion when someone comes over to julia and finds out that matrix multiplication is the same speed in julia as their old language (or slower since we default to bundling OpenBLAS instead of MKL). Hell, for people just writing simple scripts and restarting julia often, especially plotting, Julia is probably significantly slower than Python since a non-trivial amount of time will be spent compiling functions before the first time they're run.
Julia's incredibly expressive type system, macros and the crazy composability of julia code are much more impactful benefits than it's runtime performance, but these are much more vauge squishy, qualitative concepts than quantitative concepts like runtime performance.
Julia's compiled code might be super fast, but developing (or, God forbid, doing any sort of analysis) is painful because of how brutally slow the REPL/interactive environment is. Pretty much every little snippet of code you'll want to test as you write Julia feels like it takes _forever_ to run. I don't know if there's a solution for this while retaining the compiled run-time performance. I'm new to Julia (from R and Python), but I find the slowness/sluggishness of REPL to be nearly a deal breaker for me. It feels like the web back in the 1990s when you'd click a button and wait, and then click another button (or link) and wait, etc.
if you want the flexibility to do something truly bizzare with confidence, you really ought to use Julia, and don't look back.
A while back I was prototyping let's just say... unusual binary datatype representations for numbers. All i had to do was reimplement a handful of operations (+, -, x, /, one, zero) and I got everything from fourier transforms to matrix solving for free. Comparing numerical performance with standard IEEE representations was then easy, and I had confidence that my comparisons were legit, since I was literally calling the same function against both numerical types.
More recently I wanted to play around with galois fields, following Mary Wootter's impressive work, and was able to test some ideas very quickly (and trivially deploy on a supercomputer cluster) in few lines of code using Julia.
That's not a typical use case, but it's a thing. I am thinking about playing around with complex numbers (when I get some free time, which increasing seems like 'never') in deep learning, and for similar reasons Julia will be an obvious choice.
You should use Julia over Python because of the JIT compilation makes it faster, as long as you aren’t constantly writing code and running it a single time. Also as long as you are able to keep your REPL open for days at a time. It takes several minutes to load some popular plotting packages in Julia (because of the JIT), so it will take you some time to adjust your workflow to leaving your REPL open all the time.
But the Juno IDE is just as good as Jupyter notebooks or RStudio, and there are probably high quality libraries for whatever you’re doing, unless you work in one of those niche areas.
Julia JITs to native LLVM code and is really fast for a lot of use cases. For things where the JIT warming up takes less time than the full job, it can be better than Python. It was also developed with numerical computation in mind, so it was designed for that performance wise and has so many brilliant people working on the language and library. You can use macros for awesome DSLs and run on the GPU and in parallel a lot easier than Python in some cases. It has a first class package manager, great REPL and doesn't need an installer.
I didn’t find the REPL that great, because it takes forever to jit anything. My laptop is not that old and creating an array with 5 elements takes over a second. If I type a syntax error, it takes tens of seconds to produce an error. This yields a very frustrating experience and doesn’t lend itself to an effective prototyping environment. For now at least I’ll be sticking with matlab.
There are indeed frustrating lags due to JIT, but they have got better lately & are being worked on -- if I understand right this is now one of the priorities, after focusing on getting the breaking changes done before 1.0.
Here's how long a vector of 5 random numbers takes, after a cold start, on 1.1:
$ julia -e '@time rand(5)'
0.054343 seconds (121.03 k allocations: 6.219 MiB)
Yeah, this has been my experience as well. I really _want_ to like Julia. But so far the JIT experience has been exceptionally miserable--and unfortunately for me, my typical approach to development is quite interactive. I'm on 1.1.1.
That sounds really bizarre. What version of which OS are you using? I've heard that older versions of Windows (I think 7) have problems with the REPL that cause extreme latency.
I don't think the latency you're experiencing is normal.
it's faster than other scripting languages, allowing
you to have the rapid development of Python/MATLAB/R
while producing code that is as fast as C/Fortran"
Also if you have any need to generate plots & graphs - RIP Julia. I was excited to try Julia, since it seemed to integrate some of the best features of each MATLAB, R, and Python. I was truly disappointed to discover that Julia would take minutes to render the exact same plots I was generating in Octave almost instantly.
There's PackageCompiler[0] which allows to precompile the packages you need for work into a system image. This means avoiding precompilation in the REPL every time you start up Julia. That's if you start with the custom system image.
However, it's a community effort and is somewhat non-trivial to get up and running with. Once Julia gets better precompilation/binary packaging support, common workflows like plotting will improve dramatically.
to be fair, that typically only happens the first time you make your graph; replots are typically lightning fast. but this was a major frustration for me, I haven't had occasion to use it more recently, but I understand maybe things are a bit better for graphing in julialand in the last few months?
I had almost the opposite experience lately. I started doing some data analysis in Python/NumPy/Matplotlib because I figured it was mostly plotting and would be quicker in Python. It was excruciatingly slow. Partly due to wanting hidpi plots on my Mac. After switching to over to Julia, the new plots package has been complete enough to handle most use cases and only takes <5s to complete after warmup compared to matplotlib’s 30s+ in many cases. Nice benefit is that Julia types shortened up my data cleaning code significantly too. My favorite Julia plots combo has been gr in Jupiter notebooks.
this is not really an issue, as long as you ignore the julia plotting capabilities; which should have never been there anyway. You can easily dump your numbers (and functions) on a text file and gnuplot them.
Uhh no. I do a lot of data analysis and if I have to dump stuff into text files every time I want to visualize something quickly then I'm going to go mental.
Simple plots take a fraction of a second in Python/R/Matlab. I feel like many people don't realize how crucial this is. Sub-second plotting makes working with data interactive. If it takes more than 5 seconds to produce simple plots, that's no longer interactive. Imagine if your debugger took half a minute to show you the value of a variable while trying to find a complex bug. You'd start pulling your hair out.
If in Julia it takes me half a minute at least (dumping to text file, reading it in somewhere else and then plotting it), Julia is going to remain firmly in the "check this language again in 2 years time if the plotting story has become sensible yet".
It would be great if it were quicker. But right now, interactive use is pretty good, like 5ms for a simple plot. It's calling julia from the command line, and thus starting cold, which is more than 5s.
That is just a really terrible way to do things and I have no idea why you would excuse it like that. Typical workflow if you use the Plots.jl package is to wait half a minute for the package to load in the REPL and then stay in the same REPL for your workflow so that you won't have to reload the package.
What I do is wait one second for the RCall.jl package to load and then use R's ggplot2 library to plot. Works really well, especially since I am really familiar with ggplot2.
I do not use the REPL, I call julia scripts from elsewhere, and then I want to recover the data out of julia, and text files are perfectly appropriate (a few thousand numbers). Then julia plotting capabilities are suboptimal with respect to specialized plotting software like gnuplot. I would really prefer if julia did not have any plotting stuff.
That's not typical workflow for the majority of data scientists. And saying you prefer Julia not have any plotting stuff sounds really really dumb to be frank.
Agreed. As a data scientist myself, I can't imagine Julia getting much "mindshare" among us with the JIT experience it has. Perhaps we're not the real target audience for Julia? But if that's the case then adoption will likely be slow, and limited to only very niche applications and roles. For Julia to really become the next big thing (and solve the damn two language problem), it needs to be an effective solution for data scientists and machine learning engineers--and right now, it just isn't.
I do machine learning and computer vision in python, statistical analysis, plotting, and anything to do with dataframes in R, and computational stuff, network science, and almost everything else in julia. I would like to switch my data analysis stuff to julia but waiting for libraries and functions to load is just too frustrating when I'm doing things interactively. I'm hoping Julia will have a good machine learning, computer vision, and data science environment in the future and it is looking like it will. But for now, it is not an easy environment to work with in these applications and you'd need some fairly specific needs to justifiably use Julia here. But the thing is that when you do have relatively esoteric things to do in these applications, it is much easier to do them in Julia.
Startup feels instant to me. Perhaps you are talking about precompilation. In that case, it’s insignificant for most computationally intensive applications.
I don't know the difference between startup and precompilation, and I do not really care, but if I launch a julia script from the command line it is unbearably slow for no apparent reason. Octave, on the other hand, launches instantly and starts making computations.
This is understandable, because julia is not intended to be used that way. You are supposed to "live" inside the repl. However, I prefer tools that are flexible enough that can be used comfortably in non-intended ways.
I've moved all of my development from Python to Julia - The syntax is similar, but I think Julia's is more expressive and powerful once you start looking into the details. Macros, broadcasting (automatic vectorization), lambda functions, string interpolation, 1-based indexing (which weirded me out at first but I found to be more natural for most things I work on). It's been an almost wholly positive transition.
