I mean, you can almost fit all that data on one SSD. Micron's latest are 30TB. Commodity servers are available with 24 NVMe drive bays. At 7 GB/s read, across 24 drives, you can scan 40 TB in 240 seconds.
Such a server could readily fit 384 threads with dual EPYC and would be available with enough RAM to keep more than 10% of that data in cache.
Your workload absolutely fits the definition of "fits on one machine".
I am not suggesting here that you should put it on one machine. You definitely could, though.
Such a server could readily fit 384 threads with dual EPYC and would be available with enough RAM to keep more than 10% of that data in cache.
Your workload absolutely fits the definition of "fits on one machine".
I am not suggesting here that you should put it on one machine. You definitely could, though.