
Ask HN: Identifying duplicate data from a large dataset? - gerenuk
Hi,<p>We have a dataset around 150 million URLs and other meta data in ElasticSearch and looking for an efficient way to identify the duplicate URLs&#x2F;titles from our dataset. Used ElasticSearch term aggregation but it becomes very slow and returns only 10,000 URLs and most of the time it misses the URLs.<p>Currently, we have a redis with Sorted Sets, before any indexing URL, we look for the into redis set.<p>Options we have explored:<p>1. Clickhouse, storing all the URL and running aggregation etc. on it later on?
2. Storing the URLs in redis along with bloomfilter.<p>If you have worked on a similar thing, would love to hear your feedback.<p>Thanks.
======
verdverm
This is easier than deduplicating the many different URLs which have the same
content. A harder problem awaits you!

ML & basic stats

~~~
gerenuk
What approach would you be going for initially for deduplicating same urls?

~~~
verdverm
You might look at a real data processing system, like something from the
Apache projects

