I'd also start with pgvector (it's easy to switch), but the limitations around hybrid search and filtering + ANN are real and if you're doing any kind of RAG-like thing it's worth being aware of them upfront. pgvector is also an open-source project with way less manpower behind it than a bunch of venture-backed companies, so while you can expect it to pick up important features, it takes much longer (support for HNSW indices was a good example).
- Vectors are massive data wise. In our current production database they take up 95% of the memory - should they be stored separately?
- Better support for easily re-embedding, hybrid search, certain RAG workflows
- Stronger performance once you're dealing with millions of vectors.
I would still stick with PgVector until you're dealing with non trivial scale.