Very cool thanks. I am using pgvector right now. I will run into scaling issues at some point I suspect/hope.
I’ve been thinking about some setup on object storage. The data I have could be grouped per account. If you are able to compress the total index size to 1% of traditional vectors you can probably get a long way with fetching the binaries from s3 only when necessary to some sort of warm cache location.
Or just get a hetzner machine with 20TB of storage and not worry about it I guess
I’ve been thinking about some setup on object storage. The data I have could be grouped per account. If you are able to compress the total index size to 1% of traditional vectors you can probably get a long way with fetching the binaries from s3 only when necessary to some sort of warm cache location.
Or just get a hetzner machine with 20TB of storage and not worry about it I guess