On: Improving Parquet Dedupe on Hugging Face Hub

[mritchie712]

> As Hugging Face hosts nearly 11PB of datasets with Parquet files alone accounting for over 2.2PB of that storage

11PB on S3 would cost ~$250k per month / $3m per year.

HuggingFace has raised almost $400M.

Not saying it's nothing, but probably not a big deal to them (e.g. ~10 of their 400+ staff cost more).