I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get.
If you test a single array operation with NumPy then it will compare favourably to a low level implementation, but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently. For e.g. doing things like add two arrays then multiply them by another array and subtract a constant. If I put that in a C or Rust function I’d expect that to be auto-vectorized using fewer operations than what NumPy will do, and you of course also remove the overhead of the foreign function interface and control returning to the interpreter. On top of that, it’s usually trivial to drop in thread level parallelism in a lower level language, and NumPy doesn’t operate with core level parallelism out of the box for most operations, you have to use something like numexpr to achieve it.
I've personally used OpenMP + SIMD intrinsics or auto-vectorisation to great effect in scientific software and got performance >100x that of NumPy, but it's certainly not a free lunch. The question you have to ask yourself is normally “is it worth it”, and that’s something that can only be answered by profiling, looking at the overhead of maintaining the architecture, and understanding whether the increased friction with debugging and building is worth the hassle. Additionally, if you're doing linear algebra then generally if you're dropping to a lower level language you still will want to be using the same libraries that NumPy etc. uses unless you're exploiting some property of the matrix structure that can make operations more efficient in memory or compute. I worked on one codebase for e.g. where there was a block matrix structure and a hand written implementation of a solver was written because much more efficient operations could be performed with knowledge of that structure.
I've written a couple of C-extensions for a Python service that needed to run in real-time. We started out 100x slower than real-time and managed to make it slightly faster than real-time in a hectic couple of weeks.
The main bottleneck that C-extensions solved for us then was marshaling. Instead of doing something like Numpy --> custom algorithm --> Numpy --> custom algorithm in Python, we moved it all to C. This way, we decreased the number of times we had to marshal data & we reduced the amount of data that needed to be marshaled.
IIRC we gained 10-100x improved performance in all scenarios where we used C-extensions. Before doing so we tried to use Numba, as it seemed to offer a "free lunch", but we never got it to work.
The final cherry on top that let us surpass real-time speed was replacing a graph algorithm with another that had a better time complexity for larger graphs. Before doing that switch, we'd still be faster than real-time for _most_ inputs but would lag behind severely whenever we had a dense input graph come in.
It's not really meaningful to say "100x slower than real-time" or "slightly faster than real-time", "real-time" isn't some threshold of sub-second precision. It usually refers to relative (though maybe absolute wall-clock) time of execution for instructions that are typically controlling hardware that needs precise timings (ie. the time between turning this thing on and turning it back off needs to be exactly 150us).
It is meaningful, but maybe I didn't give enough context.
The service ingests data from a timespan T spanning M minutes. Before the next timespan T+1 is ingested, the service has to process the data from timespan T in <M minutes otherwise we fall behind. What I mean with "faster than real-time" is that it has to process the data faster than it is ingested.
NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scripting language can perform so well for compute intensive tasks. Of course, technically you are just stringing together highly optimized low level functions together using the Python language but the advantages of doing it that way have more often than not surprised me.
Op is better off recompiling numpy to target his specific hardware thank trying to speed it up using rust. This would be to try different underlying math implementations that are linked with numpy ( blas, mkl, eigen). Each of these haha several internal simulation frameworks to optimize not just the actual instructions but memory layout for various math kernels.
Implementing the hot path in e.g. C, compiling it into a lib an calling it using e.g. ctypes is very little effort and can yield dramatic runtime improvements / reduced bill. It's been an excellent use of my time, at least.
How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.
You use Julia and let loop fusion handle it. You get 1 allocation if you want to store the result in a new array, or zero allocations if you already have the array allocated.
numba - instead of writing in Rust, you write it in numba, which is also almost like another language. Not bad, but needs to be taken into account and is not pure numpy.
Is there an optimal solution to that for all sizes of the arrays?
E.g. I’d expect for small/medium then re-writing as a fold over the input arrays would be fastest because you’d only traverse once & everything would fit in cache.
However if the 3 arrays combined size is larger than L1 cache, I’d be strongly tempted to bet on the naïve 3 operation approach to be faster. You’d save so much time on cache line flush / reload activities.
But then if 2 arrays are larger than L3 i’d expect going back to the fold / single traversal to win again because the iterating 2 arrays at a time behaviour is no longer any different to iterating 3 at a time.
Isn't that what Theano/Aesara is for? Still based on a NumPy interface AIUI, but automagically compiling from NumPy-based to efficient code on the CPU or GPU.
e.g. comes from "exempli gratia" - roughly "for the sake of an example", or shortly "for example". In other words the "gratia" part already includes the "for", you don't have to write it again.
E.g. this is how I would use it in a sentence.
Including another for would spell out to "for for example"
> but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently
I think the bigger issue is remaining cache-friendly in the face of multiple operations. If you do the operations one at a time, you’re doing multiple passes over the data.
In any case, the approach that Polars[0] took seems like a pretty good solution, if difficult to implement well.
Polars is a dataframe library, and has a “lazy” API. This builds up a graph of operations, runs it through a query optimizer, and then executes them all at once. This allows for parallelization, optimizing expressions, and fewer passes over the data. The downside is that it’s nontrivial to implement a query optimizer.
This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it.
Thankfully there are GitHub Actions and similar tools that help with this.
I would however make the argument that if you do have a native extension but you have no binary wheel, it's better for that code to be written in Rust than C. Usually for a rust binary extension to build you just need a recent rust compiler on your machine whereas making C/C++ things to work can be a nightmare from you needing to have the right version of cmake, Cons, header files, dependency libraries and more to be installed. I have a text file with common CFLAGS and LDFLAGS I need to set to install various database drivers and more.
I tried once building a very simple Rust extension on Windows using the Anaconda Python distribution and had no luck. If I remember correctly it got stuck trying to find the VC++ compiler.
Cython extensions on the other hand work fine with the MinGW GCC compiler provided by Anaconda, although it's pretty old (version 5).
In my experience that is just the silly amount of variables you have to export to use the VC++ compiler and this is why they provide a dedicated terminal prompt for the VC++ compiler with all the right env vars loaded. I've struggled as well automating VC++ compilation in rust as a result.
Finding where & how to use an installed VS instance (or selecting one) in automated tooling is solved by the criminally unknown, MIT licensed, MS supported, redistributable, vswhere tool: https://github.com/microsoft/vswhere
There used to be a "vcvarsall.bat" script that set everything up properly without opening a separate shell session. However I recall that it moved locations on every VC++ update. Does that even exist anymore?
Generally for scientific libraries where this is common, the expectation is from many users that you'll make it available via Conda Forge/Spack/EasyBuild for easier distribution, and that anyone wanting anything else will either get a manylinux wheel targeting a lowest common denominator machine, or will have to build from source.
Having to install a Rust toolchain is still easier than trying to get C extensions to build. Before binary wheels were common on PyPi, I would either give up as soon as I encountered a C extension or switch to something like Conda to get their binaries.
Maturin wasn't powerful enough for our use cases, so we used raw setuptools-rust and cibuildwheel. Even with abi3 compat, yes it's not always easy to get the wheels exactly as you want.
And with a lack of ARM runners by default with GH actions, you'll most likely be paying for your own CI instances (or wait forever for cibuildwheel cross compilation/qemu). Also for others doing this, the rust-cache GH action saves a lot of rebuild time too.
Yeah, Python's dependency management is already abysmal, but when lower level language modules get involved its suddenly whole different level of hell. This actually put me off Python entirely. Sadly no real alternative in ML.
Rust (what I consider to be gold standard at this point), Java, C#, Clojure (piggy-backs on Java/Maven).. Hell, even Emacs-lisp has better dependency management because lack of dependency version control has established strong backwards compatibility culture within community.
Python has neither standard tooling (de facto nor de jure), has no consensus within community where everyone seems to invent their own yet-another-dependency-management-tool and no culture of preserving strong backwards compatibility (never mind the core 2->3 transition).
I've never needed to use anything beyond pip in my 9 years of programming (science, ML, back-end web development). But maybe things get bad when you move beyond those things.
But we'll agree python's dep management is leagues ahead of javascript and R, though. Right?
Yes, this. If you run something that's not in the binary cache you are an in a lot of pain because of Cargo. I have no idea how it works but it's painfully slow (is there some sort of network speed limit implemented? or is it downloading entire git history? )
This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency.
Actually Postgres does it better. Greptime only supports writing Python logic as a post cooprocessor for now, but not as an in-query function. We are trying to evolve towards the Postgres' functionality you mention.
Compiling extensions to python from another language also provide opportunities to fuse two or more operations, resulting in speed up even for cases where fast but independent implementation of operations are already available in python. And may be really worth it if that functionality is needed frequently.
I have been writing python extensions[0] using Nim language for a while now and it has been a very smooth experience every time.
I implemented a well known curve simplification algorithm in Rust, and was pleasantly surprised how easy the interaction with Python was. For packaging, setuptools_rust was great, and I too used PyO3 for the bindings. I haven't tried Rumpy yet, but it looks interesting.
I think the trouble comes when moving beyond numpy. I'm working on understanding chemistry more, and how to approximate wavefunctions for multi-particle systems. I wrote a Python script a few weeks ago that compares measured vs calculated psi'', to assess how good a trial WF is. Plotting 2/3 of the dimensions using a Matplotlib surface plot.
The surface plot brings my computer to a crawl. And, I'm not sure how to make it interactive. I gave up on it a few days ago.
Could have tried a Rust module, but ended up throwing together a custom Rust plotter in a few hours, with translated Python code. (I already have a basic WGPU-based rendering engine, with EGUI for UI). It runs much faster. They're both imperitive langs, so you can copy and paste, replacing `**` with `pow()`, `np.exp(x)` with `x.exp()` etc. And indexing with x[i][j][k] instead of vectorized numpy ops.
Ie the program compiles and runs in release mode faster than Python/numpy could perform the calcs. And the 3D graphics are smooth instead of a slideshow. And, I can make it real-time interactive since I'm using a low-level lang that integrates with GPU APIs directly. I'm not sure how feasible that would be in Python.
What did you use for your plotter? I'm in science as well and I'm shopping around for a good Rust library to use for just throwing together a quick plot of some simulation data. Currently I generate my data in Rust, write it to a numpy format, and look at it later with matplotlib, but as you've said, 3D plots usually end up looking like a slideshow.
Note that there's currently no docs or template/examples, and I'm rapidly breaking the API.
The surface plots are just meshes of a grid divided into triangles. When I need to manipulate them, I re-gen the meshes from a nested 3D array.
So, not a plotting lib at all; a flexible 3d and UI lib. Could probably made usable by others with a basic example of how you interact with the render and UI.
Wow, very cool. Personally I'm not really interested in trying to go through building something like what you've done for myself, haha.
I've come across Plotters before but last time I checked, the documentation was pretty sparse and I didn't really relish the thought of trying to parse though and tweak the examples at that time. It looks like they're gradually filling out the docs now, though, so maybe I'll give it another go.
I've used it in the past, but not recently. I remember that Diffeq lib, and it being best-in-class! Does it work for 3D (PDEs?) I remember having trouble applying it to that domain earlier, although it was outstanding for ODEs.
I think the big issue with Julia here is (other than the improving(?) JIT slowness (IIRC it was faster to compile and run a Rust program than JIT a Julia script), is I'm not sure how I'd interface with the GPU for graphics, compute shaders, and a GUI.
Julia's mathematical syntax is best-in-class; I wish more langs used something like that.
As for JIT, just today there was a PR that was merged that makes Julia cache and reuse binaries of packages (https://github.com/JuliaLang/julia/pull/47184). It won't be out until the next release of Julia, but it's a pretty major improvement to not JIT fully inferred package calls.
If you're writing a compiled extension, you can use Rust. Or Cython. Or C. Or C++. Which should you use?
TL;DR:
* If you're wrapping existing C library, I'd use Cython.
* If you're wrapping existing C++ library, I'd use PyBind11 (no personal experience, but it's based on Boost::Python, which I have happily used). Cython in theory does C++ but it's a frustrating, limited experience.
* If you're writing a tiny library and you don't know Rust, and you're not worried about memory safety, Cython is nice.
* For anything involving writing extensive new low-level code, Rust with PyO3. Memory safety _will_ bite you in the ass. Concurrency is vastly easier with Rust. You get a package manager for dependencies. You're not writing a pile of code in a language without good tooling (Cython).
To know what they do, you need the source for INPLACE_GIVE_UP_IF_NEEDED (https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f...) and PyArray_GenericInplaceBinaryFunction (I don't even know where that's coming from, it's not defined in Numpy, maybe it's part of the Python interface?).
In the end, both are unreadable in their own way. I personally prefer the Rust version above to the macrofied C version that's in Numpy but that's a matter of taste. I'd also trust the safe Rust implementations more than the C implementations because of the memory management guarantees Rust provides, though I suppose for simple operations like multiplication it'll be easier to make the program safe enough in C.
> What's more interesting is that the Rust implementation is just a factor of 1.23 slower (for large arrays) than just using Numpy
I suppose what is meant is faster (also follows from the diagram?). But it is still not a dramatic gain for many use cases. This shows how non-trivial the python performance calculus: pure python, versus numpy python, versus compiled c/c++ or rust. People who want to speed up python should really look whether numpy helps before complicating their codebase more.
But there are more benefits to those bindings besides performance so its really nice to see the expanding options
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations.
A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code.
Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which means you need to be somewhat careful if you ever relocate an install of it to a new machine
And note that for example on Apple M1 it's essentially impossible to beat an implementation that uses Apple's Accelerate library for things like matrix multiplication, because Apple uses undocumented instructions unavailable to the public in that library.
Deprecated since version 1.20: The native libraries on macOS, provided by Accelerate, are not fit for use in NumPy since they have bugs that cause wrong output under easily reproducible conditions. If the vendor fixes those bugs, the library could be reinstated, but until then users compiling for themselves should use another linear algebra library or use the built-in (but slower) default, see the next section.
> With the release of macOS 11.3, several different issues that numpy was encountering when using Accelerate Framework’s implementation of BLAS and LAPACK should be resolved.
If numpy uses runtime detection of available SIMD instructions while rust is only compiled with the x86-64 baseline (which only includes SSE2) then compiling the module with `RUSTFLAGS=-Ctarget-cpu=native` might provide some additional performance gains on number-crunchy code.
I had to do a double take. The Rust implementation is slower and harder to maintain. I recommend adding a cythonized function and a numba jitted function to the benchmark for completeness.
I'm not a Rust developer, but it seems to me that there are very real benefits to using the language. So I think the reason there are so many "I did $task in Rust!" - where $task is something usually accomplished in C, the current lingua franca of lower-level systems programming - is that people are genuinely excited to share that it really is looking possible that Rust can be used in place of C.
So it's a bit of a meme, sure, but there is a good reason for it :)
Yeah, without knowing much about Rust, I’ve decided from these types of posts that I should gravitate toward “app I use but now it’s in Rust.” I use plenty of Rust programs simply because people seem excited about it.
Yeah that's one genre of these posts - someone sharing their experience (re)writing *nix utils like cd, ls, grep etc in Rust. But as a project I think it's unrealistic to expect they'll supplant those utilities with an identical Rust-based one, there's often not much to be gained for the risk of potentially breaking a bunch of important stuff in your system by introducing a bit of an unknown quantity, and the effort to do so with any level of quality (and then maintain it) would be pretty massive.
However another genre is more interesting, to me at least - utilising existing interfaces to extend software people already use without having to make them throw out what currently works and take the risk of replacing it with a recently implemented version. In this case someone implemented a Python module in Rust, in others entire Linux device drivers have been implemented in Rust. Lower-level programming in embedded systems seems like a good application too, but I don't know how many architectures Rust can target and is officially supported on.
So to summarise, the two genres[0] I've identified are, roughly:
- I did a RIIR[1] of 50% of grep's functionality for fun
- Here is how you can accomplish a common task in C using Rust instead
As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written.
[0] - both are valid, and more exist but these are two common ones that came up
[1] - RIIR = "Rewrite It In Rust", which was a bit of a meme for a while as a bunch of devs got excited about Rust and launched projects of varying completeness reimplementing things.
> As I said, not a Rust dev and frankly I'm quite intimidated by all the rants people had about fighting with "The Borrow Checker" which sounds like a ferocious Elden Ring boss. But I am tempted by the way it allows safe (or safer) software to be written.
In my experience, those "fighting" the borrow checker are often novices misunderstanding the language model or lacking the experience necessary to effectively use it, or extremely advanced programmers running into compiler limitations and bugs. In most Rust code there's very little fighting the borrow checker.
Compare it to people saying they're fighting the compiler or fighting the interpreter when they try to multiply a string by an array and only errors come out, or trying to reassign a const value and cursing at the compiler for getting in their way. The error messages are often unhelpful and unclear (though I have to give Clang and Rust that their error messages are quite good in these days) but the core "fight" is trying to do something that doesn't make sense within the context of the programming language.
For Rust, there are additional limitations on top of what your average JS/C#/Java/Python programmer will be used to. These limitations are often also present in C (you can't just share memory between threads willy-nilly, or pass around freed pointers!) where they're classified as undefined behaviour, hopefully with a warning in the console, but in Rust you must deal with the error that's been detected.
The borrow checker is an extra constraint that's present in many modern C++ code bases as well, though the compiler lacks proper analysis support in many areas. It takes some getting used to memory ownership and the limitations and possibilities associated with it, but in many cases listening to and understanding the borrow checker will give you much better code (more correct code but also often clearer code) than ignoring the warnings like you would in C++.
For most programming languages, a beginner needs to learn 1) installing/calling the tooling 2) the core language syntax 3) the special magic features of the language and 4) how to distribute the compiled code. With Rust there's an intermediate step between 2 and 3, the borrow checker, which beginners tend to underestimate or dismiss (I certainly did) despite the warnings in any guide. It's not especially complicated, but it's an extra step other language might lack.
I recommend giving learning Rust a go. Don't be like me, don't skip the uninteresting parts of the Rust Book (https://rust-book.cs.brown.edu/, chapter 4 is what I'm referring to), you'll only find yourself getting more frustrated. Chapter 17 (fearless conspiracy) is where I fell in love with the language, but to get through it I had to go back and actually read about ownership.
OT: I quite like the concept of the borrow checker saving me from shooting myself in the foot, it's just a shame that Rust has traits, without those it might have been a nice language to use.
If you test a single array operation with NumPy then it will compare favourably to a low level implementation, but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently. For e.g. doing things like add two arrays then multiply them by another array and subtract a constant. If I put that in a C or Rust function I’d expect that to be auto-vectorized using fewer operations than what NumPy will do, and you of course also remove the overhead of the foreign function interface and control returning to the interpreter. On top of that, it’s usually trivial to drop in thread level parallelism in a lower level language, and NumPy doesn’t operate with core level parallelism out of the box for most operations, you have to use something like numexpr to achieve it.
I've personally used OpenMP + SIMD intrinsics or auto-vectorisation to great effect in scientific software and got performance >100x that of NumPy, but it's certainly not a free lunch. The question you have to ask yourself is normally “is it worth it”, and that’s something that can only be answered by profiling, looking at the overhead of maintaining the architecture, and understanding whether the increased friction with debugging and building is worth the hassle. Additionally, if you're doing linear algebra then generally if you're dropping to a lower level language you still will want to be using the same libraries that NumPy etc. uses unless you're exploiting some property of the matrix structure that can make operations more efficient in memory or compute. I worked on one codebase for e.g. where there was a block matrix structure and a hand written implementation of a solver was written because much more efficient operations could be performed with knowledge of that structure.