Hacker News new | past | comments | ask | show | jobs | submit login

I'd also mention Clojure also has a very performant 'tech.ml.dataset' equivalent to dataframes

Maybe I'm wrong, but AoS VS SoA seems like an old C false dichotomy that's been already effectively resolved

If you need to choose between the two then the answer is probably neither - use an appropriate table datastructure

There is an extra layer of abstraction and software, but I don't really see any downsides. I'd love to hear some arguments against




The downside is losing the conceptual understanding of why one or the other organization is the fastest for the data in question. From the POV of a general table data structure how does it decide which kind of memory organization is the most performant?

Looking at the examples referenced it looks like sugar over SoA so things look comfortably like normal but SoA isn't necessarily the most performant layout in memory depending on how things are being accessed.


Isn't SoA basically always more performant though? You're either traversing individual "columns" and leveraging cache locality, or you're not and then you can traverse multiple columns in parallel to rebuild the structs on the fly.

I could only see this degrade or be suboptimal if you have a very small struct that still all in-cache (like an array of 3d coordinates or something)


If you’re always accessing all the members then no. SoA shines when you want to access a different set of a few members of a struct in different contexts. Then there’s also the option of building temporary arrays of just the data you need then spinning through it which can be faster for some things. For example rendering pipelines. Even more so if you cache the results and can use them next frame due to temporal locality. There’s no silver bullet you’ve got to sit down and look at your access patterns and profile.


SOA is optimal only if you are iterating sequentially over many elements on a subset of the columns.

Really the right layout is access dependent.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: