On: Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog

[francoismassot]

So yes, currently we only support infrequent deletes for GDPR reasons mainly.

It's possible to add updates/deletes to Quickwit, but this is a lot of work, and for now, we have not prioritized this development.

Do you mind sharing your use case?

[jszymborski → francoismassot]

Oh, just run-of-the-mill full-text search of web crawls.

Some web pages update frequently, so I would need something that can handle that.

I do understand that you seem to be targeting the log use-case of Elasticsearch moreso than the "Apache Solr" use-case.

I would be in passing curious of what about the GDPR makes infrequent deletes a design choice. My understanding of GDPR is that the "right to be forgotten" aspects would if anything require deletions to be inexpensive.

[fulmicoton → jszymborski]

Web crawl can be an ok use case actually.

The idea then would be to "reindex" the world. It might seem ludicrous, but to give you an idea, indexing CommonCrawl takes about a day with 8vCPUs.

[piperswe → jszymborski]

I would assume that users requesting to be forgotten is a fairly infrequent occurrence compared to just about any other operation.

[jszymborski → piperswe]

True, but it can also require massive deletes/updates.

I suppose their are ways to structure things to minimize this, though.