> What was the biggest thing you learned while implementing this?
How much the quality of the data and resulting indexes matter. My impression based on this experience is that "RAG" might be a cohesive set of techniques, but their application to various domains likely is very domain-specific.
> Was anything surprisingly difficult?
Evaluating results is very tedious, almost by definition: you need to figure out ground truth by some mechanism and build evaluation datasets from there. To be honest, a lot of this beta was built on "vibes" only.
> Was there anything that worked better than you expected?
In terms of whether something worked better than I expected: modern embeddings are really magical. I'd previously worked with TF/IDF (a decade-or-so ago) and Doc2Vec (6-7 years ago), and while those were surprisingly useful, they really pale compared to what LLM embeddings can encode in very dense representations.
> What was the biggest thing you learned while implementing this?
How much the quality of the data and resulting indexes matter. My impression based on this experience is that "RAG" might be a cohesive set of techniques, but their application to various domains likely is very domain-specific.
> Was anything surprisingly difficult?
Evaluating results is very tedious, almost by definition: you need to figure out ground truth by some mechanism and build evaluation datasets from there. To be honest, a lot of this beta was built on "vibes" only.
> Was there anything that worked better than you expected?
In terms of whether something worked better than I expected: modern embeddings are really magical. I'd previously worked with TF/IDF (a decade-or-so ago) and Doc2Vec (6-7 years ago), and while those were surprisingly useful, they really pale compared to what LLM embeddings can encode in very dense representations.