Cool, I hadn't heard of Traycer. That does look quite similar!
Completely agree. I basically built Runner to codify the process I was already using manually with Claude Code and Gemini. A lot of developers seem to be settling on a similar workflow, so I'm betting that something like Runner or Traycer will be useful for a lot of devs.
I'll be curious to see how far the existing players like Cursor push in this direction.
I agree that's a major problem. It's not something I've solved yet. I suspect a web research sub-agent is likely what's needed, so it can pull in up-to-date docs for whatever library you need to work with.
Gemini is required for the context management sub-agent. You can use any of OpenAI, Anthropic, or Gemini for the main planning and coding agents, but GPT-5 performs the best in my experience. Claude 4 Sonnet works well too, but it's about twice as expensive.
The main thing we need to add is metadata filtering, as that's required for a lot of use cases. We're also thinking about adding hybrid search support and multi-factor ranking.
We've only done full benchmarking with the FIQA dataset, comparing minDB with Chroma. We're going to try it with Qdrant and Weaviate soon too, since they both have support for quantization, which will be a more apples-to-apples comparison with our approach.
We did test uploading and querying a Wikipedia dump, which was ~35M vectors. Query latency was around 150ms and peak memory usage was 1.5GB. We couldn't test recall, though, because we didn't have queries with ground truths.
Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.
Agreed. Retrieval performance is very dependent on the quality of the search queries. Letting the LLM generate the search queries is much more reliable than just embedding the user input. Also, no retrieval system is going to return everything needed on the first try, so using a multi-step agent approach to retrieving information is the only way I've found to get extremely high accuracy.
The queries you see and the resulting user interaction should be trained into the embedding model.
This is a foundational problem that requires your data. The way you search Etsy is different than the way you search Amazon. The queries these systems see are different and so are the desired results.
Trying to solve the problem with pretrained models is not currently realistic.
I had the same issue when searching for specific companies/products. It feels like a pretty basic vector search with no hybrid search component or reranking.
