one of the main reasons i build these ai search tools from scratch is that i can fully control the depth and breadth (and also customize loader to whatever data/sites).
and currently the web search isn't very transparent on what sites they do not have full text or just use snippets.
having computer use + websearch is definitely something very powerful (openai's deep research essentially)
if anything i would consider embeddings bit overrated, or it is safer to underrate them.
They're not the silver bullet many initially hoped for, they're not a complete replacement for simpler methods like BM25. They only have very limited "semantic understanding" (and as people throw increasingly large chunks into embedding models, the meanings can get even fuzzier)
Overly high expectations lets people believe that embeddings will retrieve exactly what they mean, and With larger top-k values and LLMs that are exceptionally good at rationalizing responses, it can be difficult to notice mismatches unless you examine the results closely.
Absolutely. Embeddings have been around a while and most people don’t realize it wasn’t until the e5 series of models from Microsoft that they even benchmarked as well as BM25 in retrieval scores, while being significantly more costly to compute.
I think sparse retrieval with cross encoders doing reranking is still significantly better than embeddings. Embedding indexes are also difficult to scale since hnsw consumes too much memory above a few million vectors and ivfpq has issues with recall.
Off the shelf embedding models definitely underpromise and overdeliver. In ten years I'd be very surprised if companies weren't fine-tuning embedding models for search based on their data in any competitive domains.
My startup (Atomic Canyon) developed embedding models for the nuclear energy space[0].
Let's just say that if you think off-the-shelf embedding models are going to work well with this kind of highly specialized content you're going to have a rough time.
What’s more interesting to me here are the calibration graphs:
• LLMs, at least GPT models, tend to overstate their confidence.
• A frequency-based approach appears to achieve calibration closer to the ideal.
This kinda passes my vibe test. That said, I wonder—rather than running 100 trials, could we approximate this by using something like a log-probability ratio? This would especially apply in cases where answers are yes or no, assuming the output spans more than one token.
If you imagine a future where LLMs get faster and cheaper even without getting better it means we'd be able to automatically repeat questions 100x and every answer could come with a pretty good confidence measure.
question: any good on-device size image embedding models?
tried https://github.com/unum-cloud/uform which i do like, especially they also support languages other than English. Any recommendations on other alternatives?
I have successfully used OpenCLIP models for embedding and similar-image search. The smallest model listed on that UForm page is 79 million parameters, so I presume that you can use other models of similar size. There are a few OpenCLIP models with 80 million or fewer parameters listed here:
When embeddings are quantized to int8 they still work very well for similarity (no differences in top 10 search on my test set). I haven't tried quantizing the models themselves.
I've been working on a small personal project similar to this and agree that replicating the overall experience provided by Perplexity.ai, or even improving it for personal use, isn't that challenging.
(The concerns of scale or cost are less significant in personal projects. Perplexity doesn't do too much planning or query expansion, nor does it dig super deep into the sources afaik)
I must say, though, that they are doing a commendable job integrating sources like YouTube and Reddit. These platforms benefit from special preprocessing and indeed add value.
The Groq demo was indeed impressive. I work with LLM alot in work, and a generation speed of 500+ tokens/s would definitely change how we use these products. (Especially considering it's an early-stage product)
But the "completely novel silicon architecture" and the "self-developed LPU" (claiming not to use GPUs)... makes me bit skeptical. After all, pure speed might be achievable through stacking computational power and model quantization. Shouldn't innovation at the GPU level be quite challenging, especially to achieve such groundbreaking speeds?
I work at Groq. We arent using GPUs at all. This is a novel hardware architecture of ours that allows this high throughput and latency. Nothing sketchy about it.
> Shouldn't innovation at the GPU level be quite challenging, especially to achieve such groundbreaking speeds?
GPUs are general purpose, a for purpose built chip that is better isn't that hard to make at all. Google didn't have to work hard at all to invent TPUs which is that idea as well, they said their first tests proved the idea worked so it didn't require anything near Nvidias scale or expertise.
The place where I work was an early adopter of LLM, having started working on it a year ago.
When I build stuff with GPT-3, especially in the earlier days, I get the strong impression that it's like we are doing machine learning without Numpy and Pandas.
with LangChain, many of the systems I have built can be done in just one or two lines, making life much easier for rest of us. I also believe that LangChain's Agent framework is underappreciated as it was pretty ahead of its time until the official ChatGPT plugins were released. (contributed to LangChain a bit too.)
Unfortunately, the documentation is lacking indeed. While I understand the need to move quickly, it is not good that some crucial concepts like Customized LLM have inadequate documentation. (Perhaps having some LLM builds on top of the repo would be more effective than documentation at this point.)
The docs are lacking but the underlying code is so simple that it's just a few clicks/rg searches away from figuring out what you need. It's all mostly ways to do string templating. IMO the ergonomics of LangChain need an overhaul; there's too many ways to do the same thing and there's too much type erasure that makes it hard to use strengths in a particular LLM. For example, it's still a pain to distinguish between using a Chat oriented LLM vs a regular completion one.
There also seems to be really poor observability in the code and performance seems to be an afterthought. I tell friends who ask about LangChain that it's great to experiment with but not something I'd put into production. Hopefully this funding helps them shore things up.
> Are you saying you'd use something else in production?
Absolutely. The Langchain interface is quite easy to build atop any old LLM, and if you're just using OpenAI then OpenAI already distirbutes clients in a bunch of languages that you can just use. Even then, you're calling a few HTTP endpoints and stuffing some data in a payload. It's really basic stuff. I'd prototype in Langchain, grab the prompts and agents I ended up using, and reimplement them in $MY_PL on $MY_PLATFORM. That's what I find so fun about these LLMs, they're just trivial to use for anyone that can ship text off to an endpoint (whether networked or local) and receive a text response.
This is what blows my mind. They raised a 10M seed on what is (no disrespect intended) a wafer thin abstraction that an experienced dev really could implement the core of in a few days. Obviously the raise was about the momentum and mindshare Langchain holds, but still. Wow.
Agreed. The text parsing for feeding prompts into an LLM and parsing a response is about as simple and straightforward as it gets in programming. It is nice to have some control over that process and know what your code is doing every step along the way. It doesn’t need to do much anyway, the LLM is doing the hard work mostly. It makes no sense to me and I’m trying to understand it, but I just can’t see the value in a black box library to interface with an LLM when it’s so easy to DIY.
I agree that their current implementation is what you said, something an experienced dev can do in a couple days. But they have the potential to really make a robust library here. The thing is, there's a lot of small things with stop sequences, token estimation, API call backoff, and generic structures that are just a pain to make yourself.
But you're right that their moat will probably be razor then. A few senior devs can get together and probably hackathon a library that's just like Langchain but much more robust. Thanks for an idea on what I'm gonna do this weekend lol.
> But you're right that their moat will probably be razor then. A few senior devs can get together and probably hackathon a library that's just like Langchain but much more robust. Thanks for an idea on what I'm gonna do this weekend lol.
Did you do it? I'm doing something similar but in Rust, for my product. It'll be open source soon enough.
I was looking through Langchain's docs and code last weekend. I'm surprised how well it is documented, actually. I thought it was fairly feature rich vis-a-vis potential chaining opportunities, but with obvious room to grow. Quite impressive, all things considered.
Well, had the community tried analyzing what could be done better? And iterating on the design?
No, the design from an old academic paper was used with a much newer model. And now everyone is just copying that.
It works, because new models are impressive. But the design is far from being elegant or particularly efficient. And now, because tons of data like that is going to be generated, we’ll be stuck with it.
Yes, relatively old. The issue is, this approach is designed to work with the “classical” language model, trained using “The Pile” methods. This particular one was Palm 540b.
So essentially you have an approach, designed to work on these models that are not really instruction following models and that truly are stochastic parrots.
The models had changed substantially since. But the approach of chaining them in this particular way stuck. And is getting copied everywhere, without much thought.
Sure. I’m just expressing my opinion that the design is suboptimal and that the level of design is literally “You are GPT-3 and you suck at math” [quote from the LangChain code base].
I don’t want to see further expansion of this. I’m not offering higher level design, because I’m not sure about safety of all of this.
But having a poor and potentially unstable design like LangChain also doesn’t contribute to it :-/
Sorry to bring up an older thread, but I was looking into LangChain recently, and I was thinking of making something similar but in other languages. Do you have any insight into what direction is better to move in for LangChain-like alternatives?
It seems pretty rare that communities get "stuck with" a framework. Frameworks are pretty fluid. Eg, Python didn't get "stuck on Django," and they didn't get "stuck on flask," and they're not "stuck on FastAPI" now - the ecosystem continued to evolve, and none of these projects even had to die for a different vision of how a framework should be organized to capture the zeitgeist. They've each got dedicated communities which are continuing to improve them.
Similarly, I expect creative hackers to pursue new approaches in the space of LLM frameworks and for some of those to catch on, and that they don't need to uproot langchain to do so.
The difference is, lots of data is being generated. And, open models in particular, are trained on it. So there is a certain level of stickiness.
An analogy of a file format is a close one. Imagine someone invents something nasty and bulky, like a twisted form of XML or something. And, simply because it is early, it catches on. It could be buggy and unstable and ugly, but it still wins, because it is early.
The call here is to try to examine LangChain design a bit closer. And maybe consider that a start from scratch is a good idea.
It's unfortunate that prettymuch all shiny AI things have horrible documentation. I see a lot of misinformation in non-researcher AI circles and I feel like it stems from that sometimes.
one of the main reasons i build these ai search tools from scratch is that i can fully control the depth and breadth (and also customize loader to whatever data/sites). and currently the web search isn't very transparent on what sites they do not have full text or just use snippets.
having computer use + websearch is definitely something very powerful (openai's deep research essentially)
reply