Apologies, it was not the technical side of it that had me musing on it. I was genuinely curious why and how I would "scroll" through that much data in a meaningful way.
And I don't mean this as a heavy criticism of the idea. I'm assuming it is useful to you. Always fun to hear about how this sort of thing is used.
You don't scroll through such a list - you use filters, pivoting etc. For end users, it's often quite comfortable to unify this in an Excel-like interface, as they're used to it. In most CRUD projects I had, users had an aversion against paged/pre-filtered displays and rather would have everything in one list where they can dynamically filter it if necessary.
Right. But that just brings us back to "how do people work with a scrolling list of 3.2m records?"
I get the point of wanting it locally to use power tools. And I get that the browser is probably capable of implementing a lot of power tools. Seems silly to insist on doing it all "in memory" on the browser, though?
That is, if the idea is you are doing pivots and filters, I don't know why a server side hit wouldn't be better for that. Similarly, when I look at something like a stock ticker for the day, I don't expect every single transaction was sent to my browser to create the graph. It /could/ be done that way, but why?
More directly to the question I had here, why and how would someone need a scroll list of every market transaction? For fine audits, I would get it, but even then I'd expect some sort of search or anomaly detection?
Still, I think if the answer is to "get it in the users hand and let them do what they will with the data," I can accept that. Goal isn't necessarily to let the users scroll the data endlessly, but for them to use any bespoke tooling they are already using.
And I don't mean this as a heavy criticism of the idea. I'm assuming it is useful to you. Always fun to hear about how this sort of thing is used.