Really well put. I hope this article can finally put to bed this very frustrating dismissal.
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb.
There is a straight forward reason here. Almost everyone, dare I say the 99%, look around and see blatant corporate-government corruption. They see an oligarchy using their wealth and power to get more of the pie each day, while escaping consequences when the misdeed.
Why should we give deference to the newest corpo kids on the block? They go hang out with Trump at what remains of the White House. Do you really think they are different from the rest of the lot?
You too see the corruption across the corporations and government that spans party lines, yea?
What are we the people going to do to remedy the situation we find ourselves in? (because they are not going to)
You're literally taking the people whom you identify as being in a giant conspiracy and saying their not-actually-human product trained to mimic their own ideas of human behaviour is safe, while disregarding all the people saying "this is not safe, we should do something about that".
This is the opposite of having faith in humanity.
I have (some) faith in humanity, that's why my p(doom) is only 10%, rather than the 95% it would be if I simply looked at, say, Elon Musk making a bot which called itself "MechaHitler" then being defended by the US executive branch when EU nations are cross about Grok making CSAM (or how slow we're being at doing anything about greenhouse gasses despite knowing about the problem for over a century), and in disregarded all the positive efforts and progress towards non-bad outcomes.
10% is still really bad.
Thinking of near-death experiences, I'm reminded of a religious joke: Man caught in a flood, waits in the branches of a tree. A dinghy comes by, he turns them away saying "god will save me". Then a motorboat, he turns them away saying "god will save me, find others who need help". Then a helicopter, same. He dies, goes to heaven, and asks god "Why didn't you save me?", to which god replies "What are you talking about, I sent two boats and a helicopter".
You want faith in humanity? P(doom)ers want to make the number smaller. Help them make the number go down, rather than denying there's something to be fixed.
Why are the researchers at the same level in China not freaking out? They have concerns and regulations, but they are not losing their minds about it? What is their perspective?
Here are a couple of videos from regular folks on the other side
There is work to be done, but it's not so close or so bad you need to let them give you anxiety. 10% is an arbitrary number with no foundation or walkthrough to how it happens. That boy wants his 15m of fame.
If you would like to help my effort improving things, please read and/or comment on this Pi issue. I'm trying to get all the harnesses to have better security posture against loss of control.
What we should not be doing is letting the big labs dictate the what and how for a technology that is (1) built on all human knowledge (2) impacts us all every day. We should all have a say in this and that will require chilling out, starting from provable facts, and having a rational debate.
Why should we trust AI company leaders? Sam Altman was quoted as saying, "We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency." Does that sound like someone who considers safety a high priority?
Those people are bat shit crazy, some random blog is not going to change my mind on that. There are Ai experts who disagree with them, who are not in the big lab friend circle / echo chamber. There are security experts who are not concerned so much that they are telling everyone humanity is doomed if we don't... give the labs complete control over who can use Ai and for what... do you want to speedrun 1984?
Sam Altman has a history, Silicon Valley has a history, the corporate elites and oligarchs have a history. Why should I believe this instance is free of that history?
Perhaps Dario is advocating for things that would amount to a ban on open weights out of fear, that does not change the effect, and I do not believe the more savvy investors do not see the benefit for that regulatory control. Open weights (real competition, not tech cartels) will undermine their financial projections. There is no way I'm going to believe the people who have been doing this for decades are not doing it here too, especially with the most corrupt administration this country has had in my life.
2. Opening with “these people are batshit” is a pretty undeniable sign that you have no coherent theory of mind for them.
3. 1984 wasn’t about regulation of superintelligence. Governments are the only ones who can say “who can use [fissile material], and for what” — has that created 1984? It started 30 years before that, so it should’ve created 1984 by 1984!
4. No one’s banning “open weights”, they’re banning superintelligence development without any of the protections we find to be obviously essential for far, far less dangerous things. Perhaps they would’ve been open to something more drastic, but post GLM-5.3 that’s kinda moot. The cat’s out of the bag.
5. Investors don’t control Anthropic, and their control over OpenAI is complex and highly unusual to say the least.
6. “Some capitalists have done X therefor all capitalists force all companies they invest in to do X” is just not sound logic.
7. Finally: that isn’t what regulatory capture looks like! It looks like revolving doors and packing boards and shady policies, not publicly campaigning for new statutes.
Scott is not an expert in AI or AI company politics. At best he can be described as a peculiar kind of cultural commentator or perhaps a political pundit.
2. I do not need a theory of mind to perceive (some of) the things they say are disconnected from reality. It also demeans our human abilities to handle these things. I'm not interested to live in a Nanny State.
3. it's about the ministry of truth, freedom of speech, Ai is closer to speech than nuclear science, which is a misleading analogy that relies on pre-supposing the Ai is super dangerous and inevitably wipe humans out.
4. there is nothing obviously essential other than humans should be staying in the loop more than they are
5. they do not need to control if they are in agreement, but that is besides the point
6. it's the same capitalists in many instances, and the patterns and trends are quite clear, especially with the biggest companies in the economy
7. Dario and Sam go to meet with the president, face to face, on a regular basis.
8. Sounds like you have too much attachment to these companies and their products to actually consider criticism in a fair light
9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
Many of your arguments rely on predeterminism, no one knows how this will play out.
I believe the best way forward is with more transparency and reproducibility, rather than gatekeepers from the oligarchy and government. We also need to put methods for proving real hummism, without sacrificing privacy, at the front of the queue.
> 9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
"Might" do.
Wanna risk it?
Even with the best will in the world, even with the best training, even with every advantage you could hope for, this is software, and the software is buggy.
We've already got loads of examples, including in the blog's comments, of it doing things other than what the user wants.
Whose speech is 3? The AI has to be a legal person in its own right if it is the AI's speech.
4. Good luck with that. The point of AI is automation and not being in the loop, which is why people were sleeping in their Tesla's while they drove as soon as it was possible, safe or not.
5. See blog
6. Name literally one company outside AI who can be used to justify the claim "safety is a plot". One.
Keeping AI in the test set just to emphasise the comparison: outside AI, I can think of literally zero examples where a company actively states their products are a danger to their customers, users, and the general public, while being sued for damages caused by their products, with no understanding of how to make them safe, as a sales pitch.
Even weapons manufacturers don't do that.
Conversely, look at NVIDIA: they don't like the safety discussion and would like everyone saying this is unsafe to stop talking.
And finally:
8. That's an incoherent position. Everyone concerned about AI safety is critical of these companies. Calling the companies' products "unsafe" is itself criticism.
I'm not against regulation, there are many places where it is needed. I agree that these things can be dangerous, but not existentially, nor today.
What I am against is oligarchy and government gatekeepers
Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
Let's take a deep breath as a society and have a rational discussion based in fact instead of hyperbole and unprovable predictions.
8. I was alluding to cognitive biases, to the other commenter. I would ask the same of you, reflect on what cognitive biases might be at play in your own interpretations. Why are you taking the side of these companies and people with such fervor?
1. Scott Alexander is the #3 most popular substack user in the "science" category, and has been following AI closely for many years on an academic level. Yes, he is a commentator; that is how media works. Yes, I think it's weird too, but :shrug:.
2. My point was that you don't understand the basics of their position. Just repeating "they're wrong" does not disabuse me of that notion. Also, anything about a "nanny state" is too far below the standards of this forum to reply to at all, sorry.
3. Superintelligence is not closer to speech than nuclear science, obviously. How often does speech lead to thousands, millions, or billions of deaths all on its own? And "their argument relies on part of their argument" isn't a rebuttal; if you read some of the papers, it is far from a presupposition.
4. I'm not sure what you're getting at here, sorry. Oh maybe this is a continuation of the above, rather than a reply to my #4? If so: yes, that's the core of the danger. You have successfully surmised one of the most serious threats.
5. Ok.
6. This just feels like a "many people are saying" argument. No, the patterns and trends are not clear: that's literally the basic question we're discussing right now.
7. That's still not what regulatory capture looks like, but also: no, they do not. Dario met him for the first time recently, and it was about intimidating him into compliance after they refused to make autonomous weapons for the pentagon. Altman has been there a few times, but A) not "on a regular basis", and B) again, that's not how regulatory capture works. The only AI CEO close to Trump is Musk, and even then it's obviously... complicated.
8. Literally the textbook definition of an ad-hominem lol. Never seen it stated so plainly and baselessly!
9. Great question! This is a large part of what's examined by the relevant literature, and they have many compelling arguments of various kinds. My favorite, layperson-targeted version is the blog post Optimality is the tiger, and agents are its teeth[1]
"Predeterminism" isn't a framework. I think the gist of what you're saying here is just "you never know for certain what's gonna happen so don't worry about it", which doesn't need rebuttal.
Yes, transparency and reproducibility sound great! It should be mandatory to conduct this research safely! We are 100% in agreement, in the end. We're just in disagreement about whether you can pattern-match this discussion to libertarian talking points from the early 2010s.
thats important! but how do the error messages compare with a decent compiler? youre gonna have more errors in your llm corrective edits loop if you dont hew to that tier of clarifying.
to be clear: youre taking this feedback wonderfully well and i appreciate that maturity :).
im just cranky with the current opportunity market and mucking with attacking a classical markov chain conjecture while i hurry up & wait to hear back from places XOR slowly build some stuff in the harness space that i hope can hit the equivalent of sublime text tier but for harnesses (is there even a market for a prosumer/pro tier harness, im not sure!)
Fair hit on "language". I keep forgetting YAML stands for "YAML Ain't Markup Language". What we built is a declarative spec written in YAML, and that's deliberate. Think of it as HTML for boards rather than a programming language. The tradeoff is readability and structure versus flexibility.
That lack of flexibility is actually a feature for one of the bigger problems we're solving: lineage, or knowing where a number came from. If the chart layer can't transform data, the logic stays in SQL, where you can version, test and audit it.
It's also a choice about who this is for. The data community already works in SQL, Jinja and YAML every day, so there's no new language to learn.
I've worked in data for 15 years and I've never heard this argument. I love it :)!
Who knows, with AI this may be the future. Visualizations were created to help us understand data. Florence Nightingale published them "to affect thro' the Eyes what we fail to convey to the public through their word-proof ears."
Often a formula better represents a situation. But I'm not sure how much impact they'd have in spreading that understanding outside of a formula-minded audience.
I worked deep in big data briefly. Billions is too big for humans so we got creative how we visualized and filtered.
AI slop has a "smell" meaning there's an abstracted data filter that results in rose or poop output. Shakespeare variable says the name doesn't matter.
So this is a fairly domain specific (dashboards) spec and we intentionally avoid getting too generic.
The raw HTML/SVG or base libraries approach may well win out, but it does make it quite hard or impossible for humans to follow along and verify for instance where the numbers on a chart came from.
I think in a future where AI's doing all that verifying (or we just trust it), the AI might still prefer to use a DSL like ours because the abstraction maintains consistency, lowers maintenance, and saves a lot of tokens.
But the most helpful bits of a structured DSL are for sure still for humans. The structured format ensures things are readable and testable. Ours also enables a generative UI, which for now at least is still a much faster way to make visual edits while working with an AI, vs always through it.
I think we're still ways out until companies will blindly trust AI that the data they pulled is correct. If a data team member sees a dashboard or chart, their first questions is "Is the SQL below this chart correct?". The easier it is to see the SQL that pulled the data, the better.
Hi HN, I'm Dave the founder of Chartio (YC'10 now Atlassian Analytics), announcing today dbt Charts, an open source YAML dialect and tool for declaring and rendering dashboards.
When making dashboards with claude or other agents, a lot of free-form artifacts are created that makes it hard to audit and scale. dbt Charts is a simple YAML dialect that declares and renders a chart (think markdown but for dashboards). Along with dbt its Apache 2.0 and launched today.
We hope this language + AI help make the BI space more open with dashboards as concise auditable code. Would love any thoughts and feedback.
We already have variables and links (which work with variables)[1]. Variables create interactive menus that update urls, queries and charts. If you have a variable `region` and you set it the url will add a get param ?region=<value>
Each chart also has a basic `link:` property can use to link to the existing or other urls with the variables set. For example
type: bar
x: revenue
y: region
link: '?region={{ region }}'
The result is a big range of flexibility, for example [2] here's an example of Conways Game of Life inside of dbt Charts.
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb.
reply