Nah, they scale. How often are you working on a large project where you're working on the WHOLE project? The agents are good enough to get the context they need, doesn't matter the project size.
I like how so many of you get offended or go on the attack acting like I'm vibe coding some small apps. I'm working at a small/medium enterprise business with about 1B in ARR. We have 100s of repos, a couple of which are our original monolithic products. There's no issue working on any of these apps. The models have no problem using gitlab mcp to get pertinent information when they need it.
Forcing an agent to ingest more source code even if it’s just a portion of the project, needlessly eats up their context window and tokens vs giving them a decent summary of it. It results in poorer performing agents that are both slower and less capable with lower quality results at higher cost (as the context window fills up, performance declines).
Also you’re telling everyone that you don’t have proof that you actually understand what your AI built. It may not bite your company now, but it will bite them one day.
This is a prime example of bad laziness. Tbf most companies like reducing as much cost as fast as possible while ignoring the potential downsides of it, so I don’t blame anyone taking on as many projects as possible.
Even WITH those docs the agent is going to review the codebase.
I've already DONE the large docs, the decision docs, the skills/plugins. I'm ripping those out now because I can see that they're garbage and they add little to zero value.
And I don't get where you think I'm telling everyone I don't have proof that I understand what my AI built. Doesn't even make sense. Because I don't have it generate tons of prose that's going to get stale? I'm okay not generating low value garbage and patting myself on the back for it.
> Even WITH those docs the agent is going to review the codebase.
But not as much of it since it has more context assuming the docs are good enough, which is my point.
> And I don't get where you think I'm telling everyone I don't have proof that I understand what my AI built.
Are you writing any of the code? Writing the tests? If not, then there isn’t any proof that you actually know the code base unless you have documentation. Even if you do know the code base one day, it’s not like you’re going to remember everything unless… you have documentation that quickly summarizes everything. Documentation isn’t just for AI. You’re likely also not going to be a company lifer so it’s also in your company’s interests for there to be documentation.
> But not as much of it since it has more context assuming the docs are good enough, which is my point.
Prove it.
Somewhat related, I DID test some of this stuff today. Plain claude, telling it to use gitlab mcp to review our codebase vs a big claude skill (for mr reviews) with a lot of details encoded in the plugin, kinda like these overly verbose and other llm generated document rube goldberg machine.
Plain claude with a simple paragraph prompt + gitlab mcp outperformed the skill by quite a bit. Less false positives, one finding the skill didn't find, ran in 4min for less than $1 vs 45min and $20 for the skill (which is basically overly verbose documentation packaged up)
The llms don't need your prose or your documentation, they can figure things out more efficiently without it.
> Are you writing any of the code? Writing the tests? If not, then there isn’t any proof that you actually know the code base unless you have documentation
The code is the documentation. You can read the code. This makes zero sense.
Your two statements are contradictory here. Docs should be good enough so the llm doesn't need to read the codebase, but it should just summarize everything? You can't have both.
I even stated in my OG post that I still keep a claude/agents.md, they're just VERY SMALL. I'm railing against the decision docs, the explosion of markdown files, and the extremely verbose code comments. All the LLM generated docs are causing friction more than anything, the llms are just good enough to work through it.
Anyone can make up a anecdote. In my case an agent was able to do a task in 10 min vs almost an hour without documentation. Citing an anecdote as “proof” isn’t good enough.
Here is my proof:
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024). On the Impacts of Contexts on Repository-Level Code Generation. arXiv. https://doi.org/10.48550/arxiv.2406.11927
Cited by: 46
Jimenez, C. E., Yang, J., Wettig, A., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv. https://doi.org/10.48550/arxiv.2310.06770
Cited by: 4998
Zhang, F., Chen, B., Zhang, Y., et al. (2023). RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2023.emnlp-main.151
Cited by: 698
> The code is the documentation.
No because it doesn’t capture the intent or the rationale why something was designed a certain way. Like why does it do some weird logic in this spot? What and why is being tested (a lot of tests are not straightforward)?
> Your two statements are contradictory here. Docs should be good enough so the llm doesn't need to read the codebase, but it should just summarize everything? You can't have both.
They are not contradictory. There’s this thing called a table of contents that summarizes large sections into a sentence or less, and links to said sections. Each section both large and small can also have a little paragraph that serves as a summary for the content to come. Sections can even have their own mini TOC. This enables an AI to be able to skip the parts it doesn’t need.
> All the LLM generated docs are causing friction
Ok, if you have a LLM generate your docs and you don’t vet it and make it your own, it doesn’t help you. Because again, how can you prove that you actually know the code base? Unless all you do is work on the same part of the same application day in and day out? Maybe the programs you’re working on is small or simple? How can you ever quickly ramp up when changing projects? This is assuming that you’re not just pairing vibe coding with manual monkey testing, or are you even validating anything when you’re not recording your goals in writing? Most of the cost in application development lies in maintenance and that is also where documentation benefits the most
I'm not asking for anecdotes. I'm asking for actual real proof. I'm not the one making extraordinary claims about how my rube goldberg machine of documents and hacks works. I'm just using the tool as designed.
You're really going to use papers from 2023 and 2024? It's 2026. The models are better these days... a lot better. Like come on man "The best-performing model, Claude 2" You're going to base your assessment on THAT?
Dude, maybe it's YOU guys working on small codebases that are worrying about "knowing the codebase" or maybe you guys just work on shit code, I don't know. This industry isn't new, there are known patterns and the models know them. It's really not hard to navigate a new codebase if you know what you're doing or if you know the framework or patterns of the language/platform. If you think you know your whole codebase though, I'd argue that it's YOU working on a small or simple app.
I'm not even understanding your line of questioning here. I don't know the codebase because I had an LLM write it and I didn't write documentation? You think writing documentation means you know the codebase? You think having your goals in writing is any different than putting it in the prompt?
> I'm not asking for anecdotes. I'm asking for actual real proof.
I gave you real proof: real research in this field on the exact subject matter WITH data.
> You're really going to use papers from 2023 and 2024?
Vs what? Nothing? Ok, prove them wrong. anecdotes aren't proof
> I'm not even understanding your line of questioning here.
I've disproven your claim 3 times. You're not understanding anything because it goes against your narrative. If you didn’t write the code and you didn’t write any documentation about the code, then you have no proof that you know anything about said code base.
> Dude, maybe it's YOU guys working on small codebases
You're projecting. You can't vibe code large projects. It's not sustainable. if you are actually vibe coding actual large projects blind, it's only a matter of time before things go really wrong.
You haven't disproven shit using research papers using Claude 2
You know it's possible to READ too, right? I don't have to write the code or documentation... not that writing documentation means anything.
I have hundreds of repos at my job, some are in the 10s of thousands of lines of code, many are in the hundreds of thousands, and we still have a couple monoliths in the millions of lines of code range.
Personal projects completely built with the LLM range in the 10s to 100s of thousands of lines.
It's sustainable. LLMs have no problem working on all these projects and working on them across repos.
But I don't know what "vibe coding actual large projects blind" is supposed to mean.
yes, and how do you prove that you both read and understand the code? saying you understand it does nothing especially when you no longer write the code.
> I have hundreds of repos at my job, some are in the 10s of thousands of lines of code, many are in the hundreds of thousands, and we still have a couple monoliths in the millions of lines of code range. Personal projects completely built with the LLM range in the 10s to 100s of thousands of lines.
Who cares because how would you be able to prove that? I can say the same thing.
> It's sustainable. LLMs have no problem working on all these projects and working on them across repos.
You have no proof of that especiall at scale. If anything all you're telling me, especially with your hard to believe anecdotes, is that you're not very experienced and you're very imaginative. It's also hard to believe anything that you've written because it makes no sense unless everyone above you isn't technical because what you're doing wouldn't fly anywhere that wasn't just a really small shop in the middle of nowhere with one or two apps.
> You haven't disproven shit using research papers using Claude 2
Compared to your anecdotes, those white papers are real work by credible professionals at respected institutions with real data that says definitively that YOU ARE WRONG. If you feel that they are wrong, feel free to prove it with specific examples and data. Right, you won't because of reasons...
I'm done humoring you. This was a complete waste of time.
> The agents are good enough to get the context they need, doesn't matter the project size.
How big are your code bases that result in this opinion? I'm working on a few million lines of code, and the AI will very very often reinvent/reimplement whole libraries within the codebase.
Only a couple of our repos hit the Ms, most are in the 100s of thousands of lines of code. Some in the 10s of thousands. But we're often working across multiple repos. Most everything is micro-services but we still have legacy monoliths that will be around forever probably. This is across hundreds of repos.
I like how so many of you get offended or go on the attack acting like I'm vibe coding some small apps. I'm working at a small/medium enterprise business with about 1B in ARR. We have 100s of repos, a couple of which are our original monolithic products. There's no issue working on any of these apps. The models have no problem using gitlab mcp to get pertinent information when they need it.