Thanks for sharing – I definitely want to plays with Aider more. My knowledge of it is limited, but IIRC Aider doesn't pull in the right files to create context for itself like Codebuff does when making updates to your codebase.
Another aspect is simplicity. I think Aider and other CLI tools tend to err towards the side of configuration and more options, and we've been very intentional with Codebuff to avoid that. Not everyone values this, surely, but our users really appreciate how simple Codebuff is in comparison.
Aider very much does pull in the right files to create context for itself. It also uses treesitter to build a repo-map and then uses that as an initial reference for everything, asking you to add the files it thinks it needs for the context. As of quite recently it also supports adding files for context in explicit read-only mode. It works extremely well.
Fundamentally, I think codegen a pretty new space and lots of people are jumping in because they see the promise. Remains to be seen what the consolidation looks like. With the rate of advancement in LLMs and codegen in particular, I wouldn't be surprised to see even more tools than we do now...
And they all converging towards one use case, and targeting one consumer base. It's still unclear for consumers, at least based on this thread what differetiate them.
I'm sure your prompting is great! It's just hard because LLMs tend to be very wordy by default. This was something we struggled with for a while, but I think we've done a good job at making Codebuff take a more minimal approach to code edits. Feel free to try it, let me know if it's still too wordy/convoluted for you.
Ah, good catch. We (read: I) introduced a bug that gave customers $500 worth of credits in a month as opposed to the $100 they paid for, and this user saw that and took advantage. Needless to say, we fixed the issue and added more copy to show to users when they _do_ exceed their thresholds. We haven't had that issue since. And of course, we ate the cost because it was our (expensive!) mistake.
The user had spent the entire weekend developing the app, and admitted that he would have been more careful to manage his Codebuff usage had it not been for this bug.
We're open to adding hard limits to accounts, so you're never charged beyond the credits you paid for. We just wanted to make sure people could pay a bit more to get to a good stopping point once they passed their limits.
> The user had spent the entire weekend developing the app, and admitted that he would have been more careful to manage his Codebuff usage had it not been for this bug.
On the flip side, there's probably useful things to learn from how he developed his app when he didn't feel the need to be careful; in a way, your $500 mistake bought you useful test data.
In my own use of Aider, I noticed I'm always worried about the costs and pay close attention to token/cost summaries it displays. Being always on my mind, this affects my use of this tool in ways I'm only beginning to uncover. Same would likely be true for Codebuff users.
Oh I see. it's an interesting story and thanks for the transparency. Might leave that out of the pitch as it's confusing and the thought of running up a $500 bill is scary and since the user ultimately didn't pay for it, seems like noise.
Have you considered a bring your own api key model?
Amazing, glad it worked well for you! I main VSCode but tried Zed in my demo video and loved the smoothness of it.
Totally understand where you're coming from, I personally use it in a terminal tab (amongst many) in any IDE I'm using. But I've been surprised to see how different many developers' workflows are from one another. Some people use it in a dedicated terminal window, others have a vim-based setup, etc.
FWIW I don't find that I'm losing good engineering habits/thought processes. Codebuff is not at the stage where I'm comfortable accepting its work without reviewing, so I catch bugs it introduces or edge cases it's missed. The main difference for me is the speed at which I can build now. Instead of fussing over exact syntax or which package does what, I can keep my focus on the broader implications of a particular architecture or nuances of components, etc.
I will admit, however, that my context switching has increased a ton, and that's probably not great. I often tell Codebuff to do something, inevitably get distracted with something else, and then come back later barely remembering the original task
Hah no, you're not alone! Candidly, this is one of the top complaints from users. We're doing a lot of prompt engineering to be safe, but we can definitely do more. But the ones who take the leap of faith have found that it speeds up their workflows tremendously. Codebuff installs the right packages, sets up environments correctly, runs their scripts, etc. It feels magical because you can stay at a high level and focus on the real problems you're trying to solve.
If you're nervous about this, I'd suggest throwing Codebuff in a Docker container or even a separate instance with just your codebase.
Genuine question: at what point does the term RAG lose its meaning? Seems like LLMs work best when they have the right context, and that context must be pulled from somewhere for the LLM. But if that's RAG, then what isn't? Do you have a take on this? Been struggling to frame all this in my head, so would love some insight.
RAG is a search step in an attempt to put relevant context into a prompt before performing inference. You are “augmenting” the prompt by “retrieving” information from a data set before giving it to an LLM to “generate” a response. The data set may be the internet, or a code base, or text files. The typical examples online uses an embedding model and a vector database for the search step, but doing a web query before inference is also RAG. Perplexity.ai is a RAG (but fairly good quality). I would argue that Codebuff’s directory tree search to find relevant files is a search step. It’s not the same as a similarity search on vector embeddings, and it’s not PageRank, but it is a search step.
Things that aren’t RAG, but are also ways to get a LLM to “know” things that it didn’t know prior:
1. Fine-tuning with your custom training data, since it modifies the model weights instead of adding context.
2. LoRA with your custom training data, since it adds a few layers on top of a foundation model.
3. Stuffing all your context into the prompt, since there is no search step being performed.
Gotcha – so broadly encompasses how we give external context to the LLM. Appreciate the extra note about vector databases, that's where I've heard this term used most, but I'm glad to know it extends beyond that. Thanks for explaining!
I think parsimo2010 gave a good definition. If you're pulling context from somewhere using some search process to include as input to the LLM, I would call that RAG.
So I would not consider something like using a system prompt (which does add context, but does not involve search) would not be RAG. Also, using an LLM to generate search terms before returning query results would not be RAG because the output of the search is not input to the LLM.
I would also probably not categorize a system similar to Codebuff that just adds the entire repository as context to be RAG since there's not really a search process involved. I could see that being a bit of a grey area though.
Thanks for being one of our early users and dealing with our bugs! I love that we can fit into so many developers' workflows and support them where _they are_, as opposed to forcing them to use us for everything.
Another aspect is simplicity. I think Aider and other CLI tools tend to err towards the side of configuration and more options, and we've been very intentional with Codebuff to avoid that. Not everyone values this, surely, but our users really appreciate how simple Codebuff is in comparison.