Ask HN: Have you hit a context ceiling with an LLM? What did you do about it?

[jMyles]

For the past few months - since the "long-context" beta feature landed in Cursor - my team has been interacting with Claude 3.5 Sonnet for a wide variety of purposes, from boring code changes to learning about each other and identify our shared goals.

We're a team of 5 musicians with varying degrees of experience in programming (from none at all up to decades of professional experience). We want to build something special to document the indices and historical anchors of bluegrass and other traditional music, and to create economies around them to do better for musicians than the labels and ticketing services have been able/willing to do.

The LLM was like having a sixth member of our team who was able to follow our team members day by day as they have evolved their core competency from guitar and mandolin to javascript and solidity. It's been a wonderful experience.

...but then, late last night, we suddenly started getting "Connection failed" for every message. Through poking and prodding in new chats, and pasting and/or attaching different amounts of the previous chat logs to them, it became obvious that we've hit a context ceiling.

We tried getting our own Anthropic API key, but indeed we hit a token rate limit immediately.

So, my question is: what's next? What needs to evolve or change so that we can have this member of our team who can help interweave the different stages of our journey with respect to programming, music theory, booking shows, etc?

How practical is it for us to write/train our own GPT for this purpose? Is that something to consider in the coming years?

[korkybuchek]

Summarize along the way, instead of attempting to use the context like a database. Or look into RAG. Or use Gemini-1.5's 1 million token window.

[jMyles → korkybuchek]

Those are great tips. The RAG thing looks doable. Is there a pre-fab solution, or do I need to use one of the APIs for the big-box providers?

My concern is: although it looks relatively easy to implement, I'm not sure I want to build a whole interface for it. I wish Cursor exposed, eg, this:

https://github.com/anthropics/anthropic-cookbook/blob/main/s...

[korkybuchek → jMyles]

It's a bear to implement well. I'd start with Gemini and see how far that context window can take you. Then I'd work on summarizing fragments of earlier conversations down to key points to include in the broader conversation. Then I probably still wouldn't try to implement RAG unless time was not an issue.

[jMyles → korkybuchek]

Gotcha; thanks for the tip.

Gemini seems to be much, much worse at the interpersonal dynamics that are important to us. It can't seem to keep track of who has what life experiences and how they might be relevant to our problem-solving, where Claude does this rather well.

I'm working with Claude now to create a summary capsule.