Imagine that the whole city has to install iron doors and window grilles, because wooden doors and glass windows are not safe enough anymore.
That's a lot of work, but is it productive in the end? Probably not.
With AI it's the same. World was fine with lax security approach before. Today it is not fine. Tomorrow we might need to prove every code line before shipping, will that be more productive? Hell, no.
> Imagine that the whole city has to install iron doors and window grilles, because wooden doors and glass windows are not safe enough anymore.
> That's a lot of work, but is it productive in the end? Probably not.
Your imaginary scenario defines the solution to be pointless. If that same city was subjected to weekly riots smashing windows & doors, and people regularly died, stopping "productive" work to install things that keep people alive and productive? That's productive.
There is a sense in which productive is being used here that is slightly subtle. Security doesn't really add anything to net output. It is, in other words, a cost. That doesn't mean it isn't valuable.
One way to think about it is that production allows you to pay for stuff. Not necessarily directly but by way of trading. It is the output you use to buy things like security.
Consider an apple farm. What does the apple farm produce to pay its bills? Apples. They could also dedicate resources to pears, &c, and that would add to their production -- to the goods they produce, the goods with which they (indirectly) purchase tractors, fence posts, &c. They could also dedicate resources to security -- more fences, &c -- and this may even be prudent. Maybe thieves are getting taller. This doesn't really add to production, though; at best, it maintains production at the status quo ante (assuming the fence is effective).
True in the sense of physical security, but I think it's not true here.
Security bugs are just bugs, as Linus likes to say. And the bugs are often also crash or correctness bugs (use-after-free for example usually causes segfaults). For the most part this actually is a productivity boon in the sense of getting higher quality goods out the other side.
To use your apple farm example, it would be like an apple farm where, if the apple rots, it's sometimes just gross but also sometimes it explodes and kills you. Would it be productive to reduce the rot rate of the bomb-apples? I think so!
It's a terminology issue. Economists use the term productivity in a certain way and they even use the term "broken window fallacy" to describe this debate.
Basically, if you accept this line of reasoning, then someone going around breaking windows and someone else going around repairing them can be seen as good because it creates jobs. But that would be nonsensical and create poverty. That's why economists focus on wealth creation not job creation.
Fixing security bugs is only needed because people exploit them. It creates work but it doesn't create wealth.
Fallacy applies only to incidental breaking of glass like child smashing window paying football.
Protecting yourself from bad actors also doesn't create wealth but it protects wealth you already have.
You know cybercrime and APT groups are not bored kids in their parents basement, there are groups that are running automated tooling to find vulnerable companies and if possible take all their wealth.
Just look at Binance of course, but if you are small company there are also people targeting those especially with AI assistance nowadays.
>Fallacy applies only to incidental breaking of glass like child smashing window paying football.
It does not. Scroll down in the given article, and you will find war as another example.
As GP said, it's a terminology issue. War spending, including defense spending, is not productive, because it does not produce wealth. It does not mean that protecting yourself from enemy countries or criminal is useless, or not worth it.
Nobody claims investing in security is a bad idea. It's just not productive, in the economic sense.
Just like securing a system is a trade-off, attacking one is too. Risk vs reward and time vs reward. Your personal blog doesn't need to be fort Knox, just patch it properly and you should be fine.
The software is better in the end but not all of it will be productive.
This why we pay cybersecurity folks, developers and business people are often living in a fantasy land where people don’t steal bikes because it is illegal to do so.
Then they have a pikachu face when someone smashes their car window to steal something worth $50 they left on front seat because “who would do that?”.
What is even better thief will most likely pawn this for $10 but for him it is effortless $10.
> If that same city was subjected to weekly riots smashing windows & doors, and people regularly died..
The point it, this was not happening before. LLMs made cheap cyber attack possible. But it didn't make the implementation of fixes correspondingly cheap. So now everyone needs to spend extra money/effort to install grills to cover where LLMs fall short.
For postgres, this absolutely makes sense though as long as people are willing to shell out the profits from LLM use to support postgres development. That is one way this could workout for everyone..
And now imagine the introduction of iron has led to crowbars that make smashing doors a lot easier. Now it is quite unreasonable to account the reduction in smashing doors to iron, when the previous improvement was also due to iron.
The physical world comparison is interesting. In the real world we have a small amount of security, but most things could be broken in to. Instead of fortifying everything, we just imprison the people breaking stuff.
Obviously this doesn't work on the internet since most hacking comes from overseas out of our control. Which makes me wonder if there is a possible future where we just block packets coming from countries that don't lock up the hackers.
Not obvious to me. I am generally a fan of Cyberspace manifesto and the free Internet, I think it's great people can reach outside their ingroup and talk to strangers and ostensible enemies.
But, that doesn't mean we should give up physical security, the guarantee that the connecting party comes from a certain physical location (state, company). I think allowing connections from anywhere by default is fundamentally flawed security architecture, for certain things.
I don't really feel like an expert on this, but perhaps residential proxy shouldn't be allowed to connect to critical infrastructure without additional authorization?
The point about a residential proxy is that you can’t tell it from a normal residential user. If a user is supposed to access the banks website, how would you filter out the North Korean? Or if a regular user is supposed to be able to do an interview with someone working at binance, how would you filter out the North Korean?
You can't tell it today, because the thing is legal. But maybe you shouldn't be allowed to misrepresent your identity when connecting to a bank. And if you have to, then there would be an additional check or the operation would be delayed.
If you're running a proxy, it's your responsibility to keep NKans out.
I am not saying close down all the Internet. But I do believe there should be tiers.
In the rest of the world making sure your product works like it is supposed to is considered table stakes. In software we somehow consider checking the parts of it we can think of 'good enough' and the rest unproductive
> In the rest of the world making sure your product works like it is supposed to is considered table stakes.
I don't think that's true.
In medicine, science often doesn't know how specific drug really is working. They just do a lot of tests and if these tests are positive enough, they ship it. I think it's good analogy. Biology is very complicated and it's out of our grasp to truly understand how various parts of it interact with each other on so many levels. But if something helps, it's stupid not to use it just because we don't fully understand it. Sometimes it bites. Sometimes it bites 30 years later. But we still follow that approach.
Software is similar. The complexity of the systems we build is enormous, when we're taking in account all the moving parts. I just read that someone emits radiowaves toggling memory bit and ex-filtrating information. How can you account for that? Probably you can't. But we still need software for good things.
Not many products are held to the same standard as software. An equivalent would be something like recalling a car model because the breaks stop working when an attacker cuts the brake line.
I think the physical world comparisons should keep in mind the kind of data a lot of these systems (eg postgres) handle. It's more like securing a bank vault than locking a home's door.
People usually don't install steel doors because most burglary is prevented in other ways. Insecure software will be exploited this way or another, if the vulnerabilities are real and the threat model is valid.
> Insecure software will be exploited this way or another, if the vulnerabilities are real and the threat model is valid.
The algorithm is: if the threat model is valid, there are vulnerabilities and you either have some software everybody knows about or a gang of hackers specifically targeting you, you should patch.
The part of the industry that profits from security theater will want you to patch no matter what, even if your business goes down in flames due to downtimes and new bugs introduced by patched components. They won't remind you to assess the threat model first.
There has to be a responsibility model for that; somebody who can say “we don’t do X even if the auditors say we should, because Y”. But I see where you are coming from, I’ve seen security theater to cost $$$$$$$ at the org I’m part of.
The reason you don't need to secure your house like a medieval castle is twofold:
(1) We live in nations of laws. The threat of state violence deters most thieves.
(2) You don't actually keep most of your valuables in your home. The stuff the thieves want is (mostly) actually stored somewhere else.
Security work absolutely is productive if it actually prevents theft. Securing your home is the wrong analogy, because it doesn't really prevent meaningful losses. Securing the bank? Different ballgame.
By this line of thought building nice and usable UIs is not productive too, because users can learn cli tools (they did, DOS) and it costs less to develop them. And all the work in TLS/HTTPS has been wasted because HTTP is enough. And ssh vs telnet.
It becomes productive only once your business stops for days because somebody brought it to a halt by breaking into the servers and deleting them.
If one thinks that "Security work is not productive."
By the way, I think that even without safety concerns seat belts are productive because they anchor the driver more firmly, so we can brake harder and turn faster (a lot of roundabouts here) and get to our destination in a shorter time. Of course this is against general road safety.
it actually is productive, because steel bar retailers and metal door installers will make a ton of cash and then 'trickle' that money into the economy.
This wave is largely bugs found in pre LLM code. I'm sure there will be a smaller wave of bugs from llm code from the sonnet 3.5 - opus 4.7 era, and peeople will always write sloppy code/use dumber models under time or cost pressure, but since Opus 4.8/GPT-5.5 xhigh LLM code stopped being intrinsically worse than human code. At this point if you're shipping llm code that's lower quality than human code that's either because you don't have the money to spend on frontier models or because you actively choose a lower quality standard because of either various pressures or just not caring. if it's the latter your human code under those same conditions would also be really bad.
Astra writes a lot of security bugs too, the upside is that if you also use it to review the output it will find a lot (most?) of them.
This is one of the least talked about and most useful features in any harness I would say, the review feature. I have never not found issues (either security or implementation) running it, no matter the code base or language.
My job is seeing a similar deluge of security work and >90% of the bugs have been in pre-2024 code, and zero of them have been bugs introduced after we were first officially allowed to use LLM codegen.
Theoretically yes, but empirically that's not the case. All models I've tried are really good at finding bugs they wrote themselves, especially in a fresh context.
This wave is largely bugs found in pre LLM code. I'm sure there will be a smaller wave of bugs from llm code from the sonnet 3.5 - opus 4.7 era, and peeople will always write sloppy code/use dumber models under time or cost pressure, but since Opus 4.8/GPT-5.5 xhigh LLM code stopped being intrinsically worse than human code. At this point shipping llm code that's lower
Software is incredibly complicated. When you get into areas like security and networking and the internet its even more complicated. We rely on layers upon layers of software and hoping they are all done perfectly with no issues. The amount of junior devs, or outsourced devs for $10 an hour working on security relevant code is very very high, and always has been. LLMs if used correctly should be raising the floor.
Unsurprising to those of us that has been screaming from the rooftops that human code reviews were barely catching anything in your typical corporate environment. The majority of the value was pointing out missing tests, because breaking bugs got past human eyeballs on a regular basis, even in very good companies.
It's not that the LLM code is that great, but that many overestimated the human code and the human review.
No it's far beyond average human code, and it's trained in RL environments on synthetic data etc now. The average human can't solve a millenium problem in mathematics but the LLMs are able to now. Anything with a verifiable output can be RLed on to make llms superhuman it seems, and the average human is not as persistent and thorough as e.g. gpt 6.1 sol at checking every edge case and hunting for bugs because that behavior is something that's just trained into the model. Frontier models perform farm above the average human at the categories of tasks they are RLed on like code and math.
> The average human can't solve a millenium problem in mathematics but the LLMs are able to now.
I think comparing the resources they spent at it would be a more apt comparison.
AKA if you tasked 150 mathematics departments to only focus about that problem, and stop doing things like teaching and researching anything else, you might get close to the amount of resources that these companies put into solving the problem to get a press release.
I've seen both of those (and others besides). I don't know how much code you've looked at but your negative directly contradicts my experience. Review that code. All of it. Be super critical because it looks good but that doesn't mean it is good. Those are two entirely different things.
Also: most tests are positive tests. Tests rarely observe the negative space: that which is there but shouldn't be there.
They definitely write off by one errors. One of the first serious work tasks I used AI for a few months ago immediately had an off by one error.
What they are really good at is writing boilerplate and reading API documentation / example code. Avoiding subtle errors is not what they're good at in my experience.
Do you mean "created" as in LLMs were used to discover existing vulnerabilities, or as in LLMs wrote unsecure code? If it's the latter, then the answer probably is "very few". Frontier models have been better than the average programmer at writing secure code that follows up-to-date best-practices for some time now.
If you consider it to be productive work, it's a massive productivity boon happening right now. Of course, product managers might disagree...