Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to go a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly.
Given that, we are working on a new file format. We hope to share our ideas/code later this year.
Honored to have your reply, wonderful view of the scene, thanks Andy.
2c remark, zero horses in this race: I was surprised how few encodings were in Nimble at release. The skeleton superficially seemed fine I guess, I don't know, but not much meat on the bones. Without nice interesting optimized encodings, the container for them doesn't feel compelling. But also starting with some inarguable clear options makes some kind of sense too, is some kind of tactic.
They claim they're trying to figure out a path to decoupling from Velox/Folly, so hopefully that can come about. I tend to believe so, godspeed.
The "implementation not specification" does seem really scary though, isn't how we usually get breakout industry-changimg successes.
I wish I had the savy to contrast lance (V2) vs nimble a little better. Both seem to be containerizing systems, allowing streams to define their own encodings. Your comment about meta-data + encodings makes me feel like there's dimensions to the puzzle I haven't identified yet (mostly after chugging VeloxCon talks).
(Thanks for everything Andy, you're doing the good work (practicing and informing). Very very excited to see ya'll's alternative!!)
There is also Vortex (https://github.com/fulcrum-so/vortex). That has modern encoding schemes that we want to use.
BtrBlocks (https://github.com/maxi-k/btrblocks) from the Germans is another Parquet alternative.
Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to go a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly.
Given that, we are working on a new file format. We hope to share our ideas/code later this year.