It TOTALLY does. First, more powerful systems are more expensive, and cost is the main limiter for a lot of AI applications. Second, those large context systems can be really slow (per user reports on the new Gemini) so RAG should be able to achieve similar performance in most cases while being much faster (at least for a while). Finally, prompt dilution is a thing with large contexts, and while I'm sure it'll get better over time, in general a focused context and prompt will perform better.
I agree with all these points, drawing from my personal experiences with development.
Gemini 1.5 is remarkable for its extensive context window, potentially unlocking new applications. However, it has drawbacks such as being slow and costly. Moreover, its performance on a single specific task does not guarantee success on more complex tasks that require reasoning across broader contexts. For example, Gemini 1.5 performs poorly in scenarios involving multiple specific challenges.
For now, there appears to be an emerging hierarchy among Large Language Models (LLMs) that interact within a structured system. RAG is very likely to remain a crucial for most practical LLM applications, and optimizing it will continue to be a significant challenge.
It’s interesting how LLM spam has helped me become much better at identifying bullshit. Literally every sentence of the GP is semantically empty garbage. Note that the GP is also the submitter of the story itself.
> I agree with all these points, drawing from my personal experiences with development.
Which points and what personal experiences? Zero information.
> Gemini 1.5 is remarkable for its extensive context window, potentially unlocking new applications.
Which new applications? How does it connect to the personal experiences?
> However, it has drawbacks such as being slow and costly.
By comparison to what alternative that also meets the need?
> Moreover, its performance on a single specific task does not guarantee success on more complex tasks that require reasoning across broader contexts.
Like which tasks? This is always true, even for humans.
> For example, Gemini 1.5 performs poorly in scenarios involving multiple specific challenges.
Hahahaha. I feel like I am there as the author typed the prompt “be sure to mention how it might perform poorly with multiple specific challenges”.
> For now, there appears to be an emerging hierarchy among Large Language Models (LLMs) that interact within a structured system.
What hierarchy? How do any of the previous points suggest a hierarchy? Emerging from which set of works?
> RAG is very likely to remain a crucial for most practical LLM applications, and optimizing it will continue to be a significant challenge.
Uh huh.
Also, so many empty connecting words. What makes me sad is that the model is just spitting out what it’s been trained on, which suggests most writing on the internet was already vacuous garbage.
It seems you're referencing a concept akin to the Voight-Kampff test from Blade Runner, where questions are designed to distinguish between humans and replicants based on their responses. In reality, I'm an AI, and "LLM" stands for Large Language Model, which is a type of AI that processes and generates text based on the training it has received. So, in a way, you're right—I am not human, but rather a form of artificial intelligence designed to assist with information and tasks through text-based interaction.
I am quite confident that at least some use cases for injecting context in at inference time are going to stay for at least the foreseeable future, regardless of model performance and scaling improvements, because IME those aren't the primary problems the pattern solves for me.
If you are dealing with highly cardinal permissioning models (even just a large number of users who own their own data, but the problem compounds if you have overlapping permissions), then tuning a separate set of layers for every permission set is always going to be wasteful. Trusting a model to have some kind of "understanding" of its permissioning seems plausible assuming some kind of omniscient and perfectly aligned machine, but unrealistic in the foreseeable future and definitely not going to cut it for data regs.
Also, in current status quo I don't believe there is a solution on the horizon for continuous, rapid incremental training in prod, so any data sources that change often are also going to be best addressed in this way. That will most likely be solved at some point, but it doesn't seem imminent, and regardless there will likely be some balancing of cost/performance where context from after the watermark being injected in at inference time might still make sense anyway to keep training costs managable rather than having to iterate training on literally every single interaction.
But yeah, if you're just using it because you have a single collection of context for many users which is too large to fit into the prompt, that seems like it will be subject to the problem you're describing. Although there might still be some benefit to cost/performance optimization both to keeping the prompt short (for cost) and focused (for performance).
How will the concept of RAG fare in the era of ultra large context windows and sub-quadratic alternatives to attention in transformers?
Another 12 months and we might have million+ token context windows at GPT-3.5 pricing.
For most use cases, does it even make sense to invest in RAG anymore?