On: Big Data Analytics Systems: What Goes Around Comes Around

[alecco]

If the data is structured, columnar databases still rule.

But if you are stuck in SV's echo chamber it's all MapReduce/Hadoop/Spark/etc.

[pacala → alecco]

How about Spark + Parquet via SQLContext / SchemaRDD?

[alecco → pacala]

The overhead of that must be ridiculous.

[pacala → alecco]

What kind of overheads should we be aware of? Spark can run on a single machine, it could be an interesting benchmark to compare against Vertica or similar products. I've seen similar systems tuned for under 1s response times, even when running in distributed mode, though they were proprietary.