Ask HN: Help me understand RAG vs. fine tuning for building a coding partner

[dchuk]

I’ve built many projects with Ruby on Rails, and still follow the framework as a hobby. I have projects I want to build with it, but as a dad with a full time demanding job, I just don’t have the time anymore to dedicate hours to coding (spare me the call outs of just figuring out how to find time).

I also have been closely following gen AI and LLM developments the last few years.

So I had a thought: what if there was a Rails coding assistant, trained on all of the popular books / courses / tutorials / gem documentation / well written open source projects?

I’m imagining something where it’s not even an IDE interface, it starts with some questions about what feature I want to build, then it starts generating the code for me to review, and I can chat it further to refine the code it’s creating for the feature. Then I can test the feature, and if it all works, I can accept the pull request basically, and move on to the next feature.

(Again, ignore the fact that building something like this also requires time commitment. Also for discussion purposes, put aside copyright concerns for a second).

My question: what would be the right architecture for this? Is RAG the best way to load up the tool with the knowledge? Or do I fine tune a model with all of the content? I can’t nail down when to use each method exactly.

I found this project: https://github.com/e-p-armstrong/augmentoolkit which sounds like what I would want to use, but then I see RAGs called out constantly, so I just don’t know.

Bonus question: let’s say I have a $10k budget. Is buying a maxed out Mac Studio a good investment for training and self hosting?

[jaggs]

As I understand it, RAG is focused more on capturing and using external resources on an ad hoc basis, where fine tuning looks to embed a specific 'behaviour' change in the model for a particular need?

[langcss]

If you are strapped for time maybe try https://www.cursor.com/ or similar offerings that do some of this for you.

Chatting with GPT4 and pasting into your editor code or pasting into a terminal some commands could also help alot.

[dchuk → langcss]

Totally agree that’s an awesome tool (and there are others that are great too). My issue though (nothing against these tools) is that they serve to accelerate writing code by automating mundane/boilerplate parts of writing code.

My ideal scenario is defining features and reviewing proposed code changes accordingly, maybe fixing little things or tweaking it when I see my original idea isn’t really the best fit, etc.

Probably overly ambitious/not fully possible yet. But it seems like a combo of RAG/fine tuning/coding agents can be strung together to invert the software development model entirely.

[menzoic]

GPT-4 is the best coding model by far and you can’t fine tune it yet. Start with GPT-4 and RAG.

Fine tune is good for changing behavior. For example you could create a dataset that makes the model directly output fully formed projects. Using RAG alone would require a bunch of prompt engineering to get the same.

[ericlewis → menzoic]

GPT-4 is fine-tunable.

https://platform.openai.com/docs/guides/fine-tuning

“ Fine-tuning for GPT-4 is in an experimental access program - eligible users can request access in the fine-tuning UI when creating a new fine-tuning job.”

You need only have fine-tuned a 3.5 model ime.

[dchuk → menzoic]

So if I’m understanding you correctly, it may make sense to fine tune a model specifically on good rails open source project examples?

[ericlewis]

You would need both. RAG will have to be part of the process if you want to generate or work on an existing project. Since it stands for Retrieval Augmented Generation. A fancy way of saying an LLM can access documents.