> He asserted that any country with its own language that did not have a sovereign LLM trained in that language was at a disadvantage as a globally trained, English-speaking LLM would not know about that country’s history, news and culture that was described in the local language.
I don’t know this is true. But whatever sounds true enough and gets funding seems to be what flies these days.
Can confirm that. Norway may have a small population, but if you live there you'll think it's truly the center of the world (aside from the US. Norwegians love America)
Not everyone is working with state secrets or user personal data (or even more closely guarded, company secrets) on a daily basis, most of what I hack on is either FOSS already, or will be, not much to keep secret here.
Obviously, if you do deal with any sort of secrets, then using local LLMs over OpenAI, Anthropic, DeepSeek or whoever is obviously preferred, and in the case of personal data of users, probably a requirement.
either this or you work on software that even if copied won't get you far since the business relies on network effects or pure networking.
Getting the source code of facebook or instagram doesn't mean you could compete with them.
I work for a company that has built relationship with event organizers over the past 10 years. The code I maintain could be written from scratch in maybe 2-3 months even though it was built over the past 10 years but besides that you have frontend / DB / hardware / logistics etc
I actually agree with you, for the most part. The code I work with actually does contain some valuable algorithms, but Im pretty sure the effort of integrating them into a larger system is pointless without the data. It’s almost like stealing half-life 2 source code without any assets.
Still, “Getting the source code of facebook or instagram doesn't mean you could compete with them.” I think to giants like that, having access to their source code could open up some very interesting loop holes for manipulating the ranking algorithms, or even security vulnerabilities.
True, haven't thought of that. However very few actual projects / companies are in a situation where the chinese GOVT would be interested to spend resources to hack your platform. For the ones that are afraid of that there's always self hosting of course
I used to work with HVAC companies, and I noticed that many of their customers mistakenly believed they were purchasing air conditioners. They didn’t consider these devices, which they connected to the internet, as computers. Despite being systems that required user names, passwords, updates, monitoring, and other maintenance, the prevailing attitude among these customers was, “This is an appliance, and why would anyone care about my air conditioner?”
All this to say, not even subject matter experts necessarily appreciate the risk involved in their work
You’re not a novice, there are a lot of us who know exactly what we are doing and see this as a huge downside. We are just being told to go faster, faster, faster lest we miss out on… something?
there's laws on the books in China that says that every company operating in China must aid and abet the Chinese government in espionage against the rest of the world. given those facts, I find it deeply troubling to be using anything coming out of China, especially a program that runs in the context of a Linux terminal on a machine that might have something important on it. I'd argue it's a back door waiting to happen, if not sooner than obviously later.
As a European I have to admit I am these days more worried about the US than China. See yesterday's article about the US government forcing Microsoft to give them lists of Dutch government officials. Utter madness. At least the Chinese mainly care about the money and power levers, the US about strange worlds of revenge and manipulation, trying to change or influence your government. E.g. which of the two countries has put crippling personal sanctions on staff of the international criminal court?
Honestly I'd love to love the US again, but basically after Obama things have just gone down and down and no soul will trust the US again in the next generation or two.
The situation you reference is related to a specific investigation by US congress requesting documents about potentially illegal censorship actions by EU officials from a specific company (microsoft). The difference is that the laws in china are broadly defined to include giving all intellectual property of anyone back to the government with no oversight, for the purposes of espionage.
The former relates to a specific investigation about potential criminal activity, the latter relates to broad illegal activity committed by the government itself unrelated to any specific case.
The US has no laws on the books forcing companies to wantonly give intellectual property and other espionage level material back to the government. If they did, no one would use cloud providers.
To avoid this, you can run your own hosted machine in a colocation facility, because in the US, people do have reduced rights when their data is controlled by a third party versus being controlled by themselves. Its the same as if the data was in your house, they would need a search warrant to obtain it, but when its at a Azure or AWS datacenter not controlled by you, your privacy rights are reduced by doing this.
I think many are trying to move away from US providers actually. FISA section 702 and the current administrations liberties taken towards international law are not helping. The trust problem is real.
Not sure I’d trust China with anything onshore. But offshore, it does seem they play by the rules, because it pragmatically serves the stability of the people. China has not started wars in the past 50 years or so. By that logic one may assume they’d not abuse the arguably broad powers over Chinese firms abroad to risk one now.
In a world where rules are increasingly less important how states use power matters more to me than how they claim to be monitored.
Besides the language barrier it’s actually also just simpler to do business with the Chinese. There are issues like censorship but they are known & can be routed around. It’s best to just ignore the US and move your business elsewhere.
so govt forcing a private coroporation being a big deal that a its on the worldwide news is more scary to you than an implicit mandate that china forces on its companies?
The four biggest (obvious) backdoor countries in the world in no particular order the United States, Israel, Russia, China. Honorable mentions, North Korea, Ukraine…
I forbidden from working on the company code with DS, but if I have a private something that looks pretty much like one of the thousands repositories put there, it doesn't matter that much.
Yeah, but it's miles better than giving Anthropic and OpenAI your data. At least Deepseek is releasing open-weight models and a lot of open-source libraries.
If you're concerned about espionage then the only solution is host the models yourself, which again, only open-weight models like Deepseek enable you to do this.
I don’t find it at all reasonable that closed source models are more efficient. The people involved had different circumstances and it naturally affects their work
If hammer companies were suddenly the most valuable international companies, and spent millions on ad campaigns and lobbying about trusting the hammer interface, then you can assume a large amount of people might trust the hammer interface
Even if your tool learns to talk and to make decisions, it's still a tool, not a person. You're the person and the one responsible for the decisions you make based on your tools.
Going back from the analogy, the problem is that we conflated software <engineers> with "coders".
A lot of people thought their job was to create code, we gave them a tool to generate a lot of code fast, and they truly think that "more code" = "more good"
Literally saw a video ad the other day which went like "I've always been cautious using Google's AI because it sometimes gets things wrong, but this time, it got it right!"
> I thought it had potential but I don't know how you'd fund it.
The same way we fund other social services here in Europe. If an individual is incapable of caring for themselves, the state is expected to care for them.
If I had a hammer robot that I told to go hammer some nails in a birdhouse and it goes "Sure, I'm on it!" then it nails a cat to the wall and says "Here's you new complete birdhouse, it's perfect in everyway and will make everyone jealous", then yes, that is a tooling issue.
That's not a good analogy then. What benefit is provided by a hammer that just tells the operator (who has eyes and can see) that there is a nail under it (and I assume to swing)?
If software engineering wants to progress past being an "art" and be considered an engineering discipline, then it should adopt methods and practices from engineering. First and foremost, one of the universal methodologies is analysis of root cause in faults, and redundancies to avoid that. e.g. the FAA has two pilots for planes, and each system is built in redundantly so if an engineer misses a bolt or rivet, the plane won't crash. intersections are designed such that there is a forcing function[0] on the behaviour of the motorists to prevent fault. Or, to take your tool analogy, nail guns are designed to be pressed against something with a decent amount of pressure before you can fire them.
All of these systems are designed around the core idea of "a human acting irrationally or improperly is not at fault" and, furthermore, that a human can have a bad day and still avoid a mistake. They all steer someone around a possible fault. Hell, the reason why we divide the road into lanes is itself a forcing function to avoid traffic collisions!
So, where is the forcing function in large language models? What part of a large language model prevents gross misuse by laymen?
I can think of examples here and there, maybe. OpenAI had to add guard rails to stop people from poisoning themselves with botulism and boron, etc. But the problem here is that the LLM is probabilistic, so there's really no guarantee that those guard rails will hold. I seem to remember there being a paper from a few months back, posted here, that show AI guardrails cannot be proven to work consistently. In that context, LLMs cannot be considered "safe" or "reliable" enough for use. Eddie Burback has a very, very good video showing an absolute worst case result of this[1], that was posted here last year. Even then, off the top of my head Angela Collier has a really, really good video demonstrating that there's an absolute plethora of people who have succumbed, in large ways or small, to the bullshit AI can spew[2].
I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability. Instead there are a sizable chunk of people saying they've stopped coding by hand entirely, and aren't even reviewing the code! i.e. They've thrown out a forcing function that existed to prevent errorenous PRs being committed! And for some bizzare reason, after about 2 decades of people talking about type safety and how we need formal verification to reduce error, everyone seems to be throwing "reduction of error" out the window!
> I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability
Development can’t be a “serious” engineering discipline because the economics of tech companies doesn’t allow for it. But this has a lot less to do about developers, and significantly more to do with the severe pressure company executives are putting on everyone to use AI, no matter what.
But let’s be honest, many companies have adopted things like root cause analysis and blameless postmortems to deal with infrastructure reliability and reducing incidents. Making systems resilient to human mistakes, making it impossible for the typo to blow up a database, etc. are considered best practices at most places I’ve worked. On the product side, I think it’s absolutely normal to make it hard for a user to take an action that would seriously mess up their account.
The core problem happens when your product idea (say, social media) has vast negative externalities which the company isn’t forced to deal with economically. Whereas in other engineering disciplines, many things are actually safety related and you could get sued over. I’m imagining pretty much anything a structural engineer or electrical engineer works on could seriously hurt or kill someone if a bad enough mistake was made.
That just doesn’t apply to software. There is a lot of “life & death” software, but it’s more niche. The reality is that 90% of what the tech industry works on is not capable of physically harming humans, and it’s not really possible to sue over the potential negative consequences of… a dev tooling startup? It’s a very, very different industry than those other engineering disciplines work in.
But, software engineering has actually been extremely successful at minimizing risk from software defects. The most likely worst software level mistake I could make could… crash my own program. It likely wouldn’t even crash the operating system since it’s isolated. That lack of trust in what other people might do is codified everywhere in software. On an iPhone, I’m downloading apps edited by tens of thousands of other engineers, at essentially no risk to myself at all.
When their precision mismatches their accuracy (or your expectations as driven by their design), just like with any other metrology tool.
Now you might say: "but the datasheet will give you the tolerances, and the manual will tell you to mind it!"
And yes, that's true. Just like how LLM providers also do: they tell you that outputs may be arbitrarily wrong, and that you should always check for mistakes.
Is this bullshit? Yes. So are metrology tools that have a mismatching precision and accuracy, need calibration, and have designs that fail to make you mind either of these, sending you to reading duty instead. Which just so happens to be a whole lot of them.
It is also absolutely not bullshit of course, because it is a fundamental limitation, just like those properties are for metrology devices. LLMs produce arbitrary natural language. Short of becoming able to perfectly read and predict the users' mind, they'll never be able to make any hard assurances, ever.
Defective devices also exist, and so do incorrect / incomplete documentation.
Agreed. I think we, as humans, like to think in terms of various metaphors when it comes to how we perceive ourselves in the world ( for example, "I am not some sort of automaton/robot" when objecting to some boss way back when ).
reply