Seems that's how most stuff we order online IS delivered. I can see the point, but maybe "unnecessary" is more a matter of perspective?
It's a shame you missed the opportunity to call this a Sparse Data Engine (SpaDE) - you would then get nice terminology about shovelling data around. On the other hand, the work around cache invalidation looks solid, and one out of two ain't bad :).
As you note in the paper, a key difference between SPDRE and things like Impulse is that SPDRE works in batch, whereas Impulse is on-demand. That means a higher up-front cost for setting up a reorganisation, but a lower cost for accessing it. Do we know how that advantages and disadvantages the two approaches in different domains?
I can imagine that for classic HPC stuff like working on matrices, batch is better. You have a matrix of structures, and you're going to access some particular field in every one of them, potentially several times. So, all of the work done during reorganisation is useful.
On the other hand, i can imagine that for searchy tasks, as in databases, access might be much sparser and harder to predict. I might have an graph of data that i want to find a path through; i expect to only touch a tiny number of the nodes in the graph, but i don't know which ones upfront. Reorganising the relevant data out of all the nodes would be a huge amount of wasted work.
The programmer interface to both approaches seems like it could be pretty similar: define a reorganisation that should exist somewhere in memory, wait for a signal that that it is ready, access it, tear it down. Does that open the door to hybrid approaches which combine on-line calculation with speculative bulk work? That would limit the interface to a lowest common denominator way to specify reorganisations; would that sacrifice too much power?
Thanks for reading! You'll have to wait for the presentation and follow-on papers for some of those answers :). If you read the Dark Bandwidth paper, there are some solutions mentioned there and in the presentation (http://www.jonathanbeard.io/slides/beard_hcpm2017.pdf) that could apply to what you suggest.