cuDF is the most impressive DataFrame implementation I've seen and have been recommending for years. The API is exceptionally close to Pandas (just a couple of different function arguments here and there), much more so than PyArrow or Modin. Throughput and energy efficiency were often 10x that of PyArrow running on a comparable TDP SotA CPU 2 years ago [1].
Correct me if I'm wrong, but with recent GPUs you should just be able to memory map a file on "disk" to the GPU address space?
Open, mmap and then import the host pointer to GPU API (Cuda or Vulkan). Paging should work as expected, but "stupid" access patterns hurt throughput.
This requires a GPU, a CPU and a kernel driver that supports it. E.g. my laptop Intel GPU has the host pointer import Vulkan API but last I checked it does not support pointers imported from mmap.
There's a profiler cell magic for Notebooks which helps identify if you run out of VRAM (it says what runs on CPU and GPU). There's an open PR to turn on low-VRAM reporting as a diagnostic.
CuDF is impressive, but getting a working setup can be a PITA (and then if you upgrade libraries...).
Personality I think it fits in the production pipeline for obvious bottlenecks on well tended configurations, using it in the r&d flow might cost diagnostic time getting and keeping it working (YMMV etc)
Does this accelerate on an M1? I know this says it is for cuda and that obviously means Nvidia GPU, but lots of ML projects have a port to Apple silicon. I would love to try this on my Mac and see what kind of acceleration my pandas tools get for free.
I wish we could commit to not conflating NVIDIA with GPU. It wouldn't hurt a soul to call it "cuDF - NVIDIA DataFrame Library." To answer your question, it will probably run on the CPU.
Jax-metal on Apple M-series GPUs is barely useable in my opinion. It's not possible to invert a matrix, for example, because Apple has not implemented the necessary triangular solve operator. It's also not possible to sample points from a normal distribution, because the Cholesky decomposition operator is not yet implemented. Apple hasn't responsed to both of these issues for the past year. It's difficult to take a numerical computing framework seriously if one cannot invert a matrix.
I should have been more precise to say that I wondered if there was anyone trying to port it to the M1. It seems like every other ML project has some pull request trying that. Or, perhaps there is another project that is attempting to do the same thing for pandas on the M1. It's a noble goal.
How does this compare to duckdb/polars? I wonder if GPU based compute engine is a good idea. GPU memory is expensive and limited. The bandwidth between GPU and main memory isn't very much either.
The same group (Nvidia/Rapids) is working on a similar project but with Polars API compatibility instead of Pandas. It seems to be quite far from completion, though.
I've been watching cuda since it's introduction and Polars since I had an intern porting our Pandas code there a couple years ago but I had no idea Polars would go this far, this fast!
This and Rapids.ai is the single reason that NVIDIA is the leader in AI.
They made GPU processing at scale accessible to everyone, I have been a long term user of Rapids and found that even as a data engineer I can do things on an old consumer GPU that would otherwise require a 20+ node cluster to do in the same time.
Polars is great. Most of my Python work involves wrangling data with pandas and polars was integral in getting me out of rust tutorial city and into rust production.
This is actually a good callout. While pipeline speedups of transforms is hugely important, lots of other fundamental python tools for viz, model examination, etc are not built on a different foundation and not optimized by pandas improvements.
I get it: some of these are legacy, others are hand optimized python since default pandas is so slow. But I'm hoping that, over time, we'll improve the runtime of the other stages of analysis too.
[1]: https://www.unum.cloud/blog/2022-09-20-pandas