On: Understanding Parquet, Iceberg and Data Lakehouses

[jamesblonde]

I agree with all of this. Databricks are also holding back features from open source Delta (like bloom filters), which is their right. But then you can't claim it is a community-driven open format, unless it is an animal farm version of that, where one of the versions is the Pig (some are more equal than others).

[hacful-tonteg → jamesblonde]

Databricks has a lot of nice closed-sourced components, e.g., Unity Catalog, Delta Live Tables and Photon (a C++ implementation of Spark).

Delta itself seems fairly open-source: https://github.com/orgs/delta-io/projects/10/views/1 and hopefully someone will implement Liquid Clustering!