On: Rayforce

[tosh]

afaiu morsel-driven means the workload gets turned into 'smallish' chunks (morsels)

instead of having to pre-allocate upfront (e.g. 4 nodes get 1/4 each) it is more granular and dynamic

a worker that's "done" can request another morsel

pragmatic approach because nodes might not all be equally fast (cache, cpu frequency, throttling, …) and also some morsel workloads take longer than others depending on the values they contain and what kind of work needs to get done

so this approach tends to balance out nicely

I'm sure someone else can explain it better / correct me (please do!)

[noelwelsh → tosh]

When I read up, it sounded like the same idea as work-stealing to me. Not surprising that different fields come up with the same idea under different terminology.

[adsharma → tosh]

DuckDB and LadybugDB use the same terminology to describe internals.

[hetoku → tosh]

Exactly!