Didn't the Postgres team just have to stop a release because the agents found a slew of bugs in human code from domain experts that "understood and verified it"?
Perhaps I'm being overly generous to the parent's point, but it stands to reason that we make a lot of choices of where and how much trust we give away. This is just another one of those points.
I would say that a calculator and a database like postgres differ quite significantly, though. How many PRs are merged per month on your average calculator? How many lines of code is in that firmware?
Trusting that a calculator works is like trusting that there will be another day tomorrow, while trusting the code of an LLM is like trusting that the current weather will hold for the next hour.
Also, LLM finding bugs in human written code doesn't mean it wouldn't find much more in LLM generated code, and spend much more money doing so. That is, of course, after ruling out the false positives (of which there are a few, judging by the PRs I've received on my open source projects)
I think you're getting caught up on a hyperbolic example. I already tried to clarify there's a gradient, so I don't know what you want from this discussion?
And that some one can be an LLM. Does your manager at your job review every line of code you wrote or trust you based on your competence and reviews only a compressed portion?
Frankly, these discussions are tiring because there's a false notion that humans have to be in the loop 100% of the time. This is untrue. LLMs are accurate enough that they can be left to do things on their own and human intervention is needed only for out of distribution instances.
> Does your manager at your job review every line of code you wrote or trust you based on your competence and reviews only a compressed portion?
The manager always has a very good stick: “Do a good job or you will be fired”. And me wanting food is well aligned to not produce buggy code. Or make my job harder in the future by writing unmaintainable code.
> Frankly, these discussions are tiring because there's a false notion that humans have to be in the loop 100% of the time
The code is written to process data. Someone needs to be aware of how it’s processed, so that when malformed data appears, we can collectively find where it happens. You can always play around with your own data (if regulation allows) but you need to respect your customers’ data. Not looking at the code jis just negligence.
> The manager always has a very good stick: “Do a good job or you will be fired”. And me wanting food is well aligned to not produce buggy code. Or make my job harder in the future by writing unmaintainable code.
You are welcome to not use the model if it makes mistakes. Its weird that you are cooking up some edge case scenario to make your argument strong - everyone is incentivised to make things work well. Anthropic wants their models to perform well because that's why you will buy it.
> Someone needs to be aware of how it’s processed, so that when malformed data appears
Eh no. You need to know about only a compressed part of it, not 100%. Do you really claim that you need to know 100% of the code? This is not true today and certainly won't be true in the future. Hardly any code will be looked at by humans in the same way my manager doesn't look at my code.
> Anthropic wants their models to perform well because that's why you will buy it.
What Anthropic wants and what happens in reality are two different things. But with code I write, I do control the process.
> Eh no. You need to know about only a compressed part of it, not 100%. Do you really claim that you need to know 100% of the code?
I know the code I write, and I know how it interfaces with external modules. When those interfaces don’t function as described, I look at the code if I have access to it. When I don’t, the party in control has always good incentives to produce good documentation. That knowledge is usually good for some time (years even)
> my manager doesn't look at my code
Because that’s not his responsibility, just like he’s not responsible for bugs and crashes. He’s responsible for the project and the team and you can be pretty sure that he’s looking at all at the information about those. Imagine if his own manager ask him about some details and he answers: “I don’t know, Claude is doing it”.
