The maths community is now in the antithesis phase, synthesis will take a while ;)
Lee Sedol said in an interview that "losing to AI, in a sense, meant my entire world was collapsing. ... I could no longer enjoy the game. So I retired", and I think there will be folks in the mathematical community who would feel the same when the solutions pages to hard problems are suddenly available.
But on the other hand, people learned a lot from chess engines. After decades of chess computers beating humans, there was still a renewed interest in watching Leela beat Stockfish, with many people trying to understand the strategy Leela used.
If your happiness comes from grinding on a problem and making progress, the prospect of having to dig through a corpus of AI-generated proofs might be hard to swallow. But if you're willing to do that, you will still find beautiful things that only so many people can truly appreciate.
Chess is a fun game. That's why it's been around for 1000+ years.
There was a renaissance during Covid and due to 'The Queen's Gambit' where it gained much more mainstream popularity, but... Chess AI was already far far (like 1000+ Elo) ahead of human players at that point.
The thing is... chess is humans playing (communicating) with humans and that's what keeps it interesting. Check out the view counts of chess AI tourneys vs. human tourneys.
Chess is kept afloat by chess players, not by billionaires. If all the billionaire backers stopped sponsoring tournaments, people like me would still play, still pay for chess club memberships, still pay entry fees for tournaments, and still buy chess books, and so on.
I think the parent comment meant professional, high-level chess. The kind people get played to play, not just do for a hobby. That's absolutely on life support.
I'm not sure what the equivalent would look like in the math field, but it probably involves a lot of mathematicians losing their jobs and the quality of human-produced math decreasing overall.
The quality of the math in general would be fine, since in this scenario cpus will keep producing it. The quality of cpu-cpu chess games is quite high, beyond human understanding in many cases.
Chess is a weird example because it doesn't really have any utility beyond itself. Even pure math sometimes ends up having use in the strangest places. Although if no one understands the frontier math (because no one is getting paid to), I'm not sure it even matters what the quality of the cpu math is?
It's a bit like a tree falling in a forest. If an LLM proves a theorem but no one understands it, did it make a sound?
"Although if no one understands the frontier math (because no one is getting paid to), I'm not sure it even matters what the quality of the cpu math is?"
Presumably AI will be connect the dots to the applications. As the declaration says, this isn't just about math. Human understanding is losing economic value. You can understand stuff on your own time, I guess.
The standard justification for pure math to holders of purse-strings is something like "it might lead to a useful application down the road, like crypto, who knows". That looks pretty inefficient now. We have to entertain the possibility that AI can develop the math needed for any application we put to it. Eg if number theory didn't exist, we could have asked AI for a way to transit messages securely and it would maybe come up with fermats little theorem as part of its solution or maybe come up with an approach we can't conceive of right now seeing as most of us are constrained to available number theory. Like how in the last year when I give an LLM a programming project I see it doesnt even bother with of the many software libraries I and others have written and just codes up the calls it needs on the fly or finds some other ad hoc solution.
> It's a bit like a tree falling in a forest. If an LLM proves a theorem but no one understands it, did it make a sound?
But in future most proofs will be for consumption by other AI models in the pursuit of yet other proofs.
It's kind of surprising so many mathematicians act surprised by this given this was clearly where automated proof assistants would lead. I guess they assumed they'd always be the ones guiding them.
It's going to be hard to compete with something that has access to all of math at once and can find connections between elements that appear unrelated to humans.
And at some point AI will start suggesting - or doing - physical experiments.
Well if it proves useless presumably they'd stop doing it.
But if AI is to recursively self improve understanding and evolving its own foundations, which are clearly mathematical, is essential. There is no need for humans to grasp what is going on in that loop.
Presumably some of the proofs will have applications beneficial to humans beyond impressing other mathematicians, and AI will surface them, or use them directly.
I mean, was the point of math ever just because some humans enjoyed doing it? Even though a lot of it is theoretical, there's been all sorts of useful things that have come out of it as well due to an improved understanding of the universe through new ways of thinking about it. If it got to the point where no human could understand it and there were no ways to actually use it, I don't think anyone would bother having their computers doing it at all.
What would stop you from doing that if AI happened to be doing a bunch of different or more advanced stuff? I assume that the fact that there are humans doing other or more advanced stuff doesn't dissuade you from doing it for fun?
Yeah, that's kind of my point. I would have to imagine that the people investing in AI doing math (using "investing" loosely, not just financial but hardware/energy/opportunity cost as well, which may or may not seem roughly equivalent to money depending on your viewpoint) would just stop unless it remained possible to apply in some way or was understandable by humans.
I would argue people getting paid to play chess was a short lived phenomenon anyway if you put it in context. The transition there is less related to the introduction of chess engines and more related to the shift in the media landscape.
When there's a billion people playing something, money will never be an issue for those at the top. Even things like chess.com was able to sponsor a tournament with a million dollar prize pool.
Also I'd argue that chess's utility is ultimately the same as pure math, particularly in esoteric fields. These things are highly unlikely to ever lead to any sort of real world breakthrough or application. The main benefit is an outlet for human logic, creativity, and exploration - which significant self improvement possible along the journey for players.
Though I think even that's probably too socially utilitarian. I think ultimately the 'real' drive is the same in both fields - it's fun and personally rewarding.
> Though I think even that's probably too socially utilitarian. I think ultimately the 'real' drive is the same in both fields - it's fun and personally rewarding.
That's fine, and no amount of machine excellence will keep you from enjoying recreational chess or recreational math.
Ah but it's enjoyable for billionaire and pauper alike. And the former tends to enjoy sponsoring it for the sake of seeing and promoting the bounds of human capability.
Mathematics is more than establishing arbitrary facts (although some look like curiosities), it's also defining what interesting research directions are and establishing common language/notation. I think that will stay relevant?
People become interested in things when they become invested in it personally, because they've contributed to it. So I don't think it will stay relevant...
In theory you can automate finding interesting research directions by identifying conjectures with many dependencies. And notation has never been mathematicians' forte, with them trying to cram the entirety of universe into single letters.
Why? How do you define interesting research directions? That used to be defined by testing the limits of human understanding i.e. some people can't figure something out. AI might have very different ideas about what is interesting and I am not sure what humans would get out of putting years into understanding AI proofs for what? What are we doing at that point? Like if you spend years understanding some AI proof of theorem 123456, why is that meaningful? I am actually asking why you think defining interesting research directions will stay relevant. In my opinion, people spend years acquiring knowledge so they can work on problems which is separate.
There's two points about this I am assuming 1) Mathematics actually has a significant subjectivity to it and is community oriented and not just climbing a never ending list of theorems that exists in the universe 2) A lot of mathematical research work is inside of a subfield and isn't directly motivated by applications. Sometimes it is but e.g. people don't work on obscure theorems about elliptic curves because of a dire need for that but more because the community found it interesting.
Does your "one" only contain humans or does it also contain other AI systems. AI math is not a single monolithic thing, but a distributed one. I see value in sharing proofs even among just AI.
Also, you learn to be a better chess player by... playing better players. The widespread availability of chess engines has made flawless opponents available to every player.
If your goals are understanding the game, self improvement, building thinking skills-- this is the best chess has ever been. It's only if your goal is to beat every opponent you can find that chess is in a bad place.
> Playing stockfish is like playing tennis against the wall (for untitled players at least).
It's the same for Magnus Carlsen. Even with Queen odds, Stockfish is literally unbeatable for the best players in the world. It's just too strong at evaluating all kinds of random tangent moves (and ensuing positional advantage) which no human player can possibly pay attention due to the time required.
Stockfish vs any human is like Carlsen vs other players by about 3-5 orders of magnitude[0]. It's that stark.
[0] A wild pun appears.
EDIT: To avoid having to respond to each responder, fair comments about Queen odds. Maybe I was thinking Rook odds? Also, I kinda lumped Stockfish in with all the other engines, but I realize there are other engines with different properties ofc.
Well, that's actually not true at all. Stockfish is not a very good odds player and at queen odds is easily beatable even by bad players like me. It will just trade down into more trivial and easier to win positions that it perceives as "less bad", since everything is super-losing anyway when you start down a queen.
Leela odds networks, on the other hand, are an entirely different beast. I cannot beat Leela queen odds, much less rook or minor piece odds, and even GMs struggle against Leela knight odds.
Without odds though, yeah, Stockfish is just incomprehensibly strong by human standards. All top chess engines are, but Stockfish moreso.
No worries, I know stockfish is unbeatable by humans.
But sometimes, these GMs can flag it, which counts as a win (especially when it's proxied by a cheater). Sometimes they can also explain the idea that cost them the game, so they've learned something maybe.
Whereas us scrubs literally cannot do anything at all for reasons completely beyond our understanding.
> Whereas us scrubs literally cannot do anything at all for reasons completely beyond our understanding
Computer moves are typically much more concrete than human moves: a human will play based on pattern matching ("intuition") and can only make explicit calculation of a small fraction of possibilities, after which decisions are guided by guesswork. The computers are unbeatable in practice because they can calculate concretely in seconds what might take an expert human long intensive study to notice, and they don't make the same kinds of oversights humans can make.
But if you stop and explore a particular position for an extended time, and if you have an intermediate level of chess skill, you too can probably often (usually?) figure out why it's doing something. Sometimes understanding the computer's reasons takes searching multiple branches of a tree several unlikely looking moves deep, but the collection of threats the computer was preemptively thwarting, traps it was setting, etc. are comprehensible to humans with enough effort, especially in games between the computer and a human.
The frustrating thing about playing against the computer is that it notices and thwarts every plan you might come up with, before you make up the plan yourself, and it doesn't make (human-apparent) mistakes, so the game ends up feeling hopeless. Nothing you try works on it, and if your idea is even slightly inaccurate it will be exploited.
You put it better than I could. The lesson of "in this exact position you can kick the pieces for 7 moves to get a fork, so instead you should play a4" is not something that I can implement into my games
I mean, I'm not a great player but I've learned a lot from working through games with stockfish using a git repo and a small script that lets me rewind to different moves and try different approaches. It may not explain its moves but if you're thinking through what's happened you can usually debug your game anyway.
I mean, this is the problem with analogies and trying to use them to prove things, right? People working through problems from an analysis book with their friend (or an LLM) is not the same as research mathematics. People playing in a chess club is not the same as what makes for a good chess tournament. Lumping everything together is just making this branch of the conversation less relevant.
I really hate this overly condescending takes. First of all, what do you know about the internals of math research that allows you to speak with so much confidence. Second, you're not even addressing the issues raised by the letter! This is not about "oh they made a bunch of problems easier". There are huge economical interest behind: who owns and has access to models? are these companies interested in developing research or they just grind PR stunts without worrying about externalities in how research is actually conducted? Etc etc.
If it's worth anything: I have a PhD in (theoretical) mathematics and I entirely stand by stabbles' comment.
There is a real, undeniable possibility of AI becoming better at mathematics in the same way that it became better at chess and Go, and in such a scenario, one may expect the community's response to be comparable.
It makes no sense to compare mathematics with chess. Chess is a sport. No one is interested in watching two machines compete. Chess doesn't have a practical impact. Etc. What you seem to suggest is that AI will be able to completely (or at least in a great part) replace mathematicians. It could be the case in the future, but no one knows right now, and more importantly: tech companies don't even think about it! they don't think on the externalities.
Does mathematics still have a practical impact without humans in the loop? I don't think there's one single answer to that question, but I think it's worth considering exactly what that impact may be.
Tech companies are as much the topic of this post as AI, I think that's the immediacy.
To a significant extent, the pursuit of mathematics research is a pursuit of human understanding of mathematics, without knowing where it might lead, or whether it might lead anywhere at all. I don't see how the motivation for that goes away on its own, but the institution supporting it is certainly threatened by the potential loss of grant money and graduate student applications.
> Does mathematics still have a practical impact without humans in the loop?
There's a single answer to that question: yes, math very much has an impact without humans in the loop.
Math has a lot of applications, and those applications don't care whether eg the new faster matrix multiplication algorithm was found and proven correct by a machine or a meatbag.
We're way into diminishing returns in matrix multiplication, and being clever about ALUs and cache layout is likely to dwarf any asymptotic improvements you're going to find. Any better examples of improvements in the last decade?
The main situation I can think of where better calculations have a really visible effect is video and image compression, and that stuff is very far away from mathematical proof territory.
Wow that's a terrible paraphrase! I asked about algorithms where recent/ongoing improvements were important and they already replied with a list.
If you tell someone they can't use a matrix multiplication with better asymptotic performance than n^2.3755 from 1990 they're going to shrug and not care.
As the US has offshored manufacturing, the number of patents issued for those processes has fallen. Innovation occurs where the foundational understanding is applied; they arise from a desire to do the required work more efficiently.
Similarly in math, attempting to solve a problem leads to new questions. IF you actually do the work.
You need to have done enough of the work to know what the correct next question are, or you need to rely on AI for everything.
so you're 100% sure that AI will solve every intellectual problem in the future, since as long as that's not the case, it's us who need to ask the questions. I don't know man, I wouldn't bet on that. What if we end up being wrong and then there is not research community?
I don't understand what you're proposing then. We can ask AI to solve practical problems of interest (making a computer faster for instance), but I'd say no one really believes that this is an unlimited resource. At some point we'll reach a plateau, and then AI will be an important tool but to continue advancing we'll need someone to ask the right questions.
What is 'asking the right questions' supposed to mean?
I am saying that applications supplied and supply an inexhaustible amount of good problems and questions to consider. Purely theoretical concerns also supply some questions, but even if that well dries up for some reason, applications persist.
In other videos he's called out the influence that this and similar games have had on human players in recent years, particularly around square denial and thorn pawn strategies.
Applications don't care whether the math was proven and understood by humans or computers. Your algorithm will get faster no matter where the insight came from.
In math, the journey is often more important than the destination. The process of developing a proof may uncover new mathematical techniques, some of which may have practical applications in other fields. Even an attempt that ends up as a dead end towards the intended proof could produce something useful in a difderent area. But if you just get the proof directly, you miss other discoveries you could have made along the way.
Take the Navier-Stoke problem for example. Knowing that there are solutions that "blow up" probably doesn't have a lot of practical applications. Such solutions couldn't happen in a real system. But the process of finding that proof could result in increased understanding of how turbulence works, or new techniques for solving non-linear partial differential equations (which has a lot of applications in science and engineering).
> In math, the journey is often more important than the destination. The process of developing a proof may uncover new mathematical techniques, some of which may have practical applications in other fields.
Sure. And AIs can use ideas from AI published proofs in one domain to inspire other domains just fine. Nothing changes here.
Applications do not care about 99.999% of theoretical math production anyway. And especially most of the big results in theoretical math nowadays are really inconsequential in applications.
Applications don't care about Navier Stokes, yes. But they care about eg proving crytographics secure, or proving that your algorithm doesn't blow up under adversarial input.
Formal verification, cryptography and the like is far from what the vast majority of theoretical mathematicians are doing (if those who do them even see themselves as that vs computer scientists or applied mathematicians) especially when it has to do with specific, production systems, and there are not many other examples like this in general outside compsci and statistics. Moreover, I can imagine that these fields will actually flourish more now that AI can make verification and proofs more viable in scale. But even much theoretical work related to cryptography etc is often not very applicable in itself.
to be honest it is difficult to discuss with someone who doesn't even try to understand the basics of basic science (and how it compares with _applied_ sicence), yet talks with so much confidence. even the solution to navier stokes won't have an immediate practical effect...
> there's also plenty of problems whose solutions will have practical effects, some even immediate.
Sure. But do you know which ones they are? Or do we discover later that they were valuable?
Your argument would be 100x more convincing if you gave an example.
I will try: a super-compressor that made my 100Mb web app into a 5 kb binary bundle would immediately speed up my work. Can/will AI move human understanding or machine capabilities on this front?
A browser without security vulnerabilities would be wonderful. I think LLMs are already helping with this a lot, but a lot of complexity remains.
A right to privacy in society would be amazing (see the UN Declararion of Human Rights). AI is eroding this.
So I tried but I’m not very impressed with my list. Do you have one?
> A right to privacy in society would be amazing (see the UN Declararion of Human Rights). AI is eroding this.
This has nothing to do with mathematics.
> So I tried but I’m not very impressed with my list. Do you have one?
Look into operations research. Or narrower, you can look at improvements in linear programming solvers and mixed integer linear programming.
(These are examples of areas that have seen mathematical improvements in applications recently. I don't think good AI has been around for long enough to contribute much to progress there, yet.)
And this is exactly the point of the Statement. The process is more important than the solution itself. Most problems in mathematics don’t have immediate value or applications to the real world.
AI’s solutions are like the answers section to practice problems at the back of a textbook. Answer is 42, so what?You have to attempt the problem yourself, that’s the whole point of the exercise.
As an engineer I’m happy to use AI for math. If I publish a paper that way, very common these days, I think it’s still
problematic. My paper would include something I didn’t come up with and I don’t really understand.
I think this is a good time to properly discuss these things because AI is coming for everything. Mathematics and Software were just the first two.
You don't seem to have grasped Terry Tao's (and others') criticisms. Basically they are saying that the advancement you get is illusory. Most of the time, it doesn't give you any new capabilities or deep understanding, instead you get an inhibiting of human exploration and ensuing expansion of our real understanding in that particular (sub)field. It's non-intuitive, since from a purely logical standpoint you've only added another set of known truths to the ones we already knew about before. The issue only becomes apparent when one considers the larger context of human collective truth and meaning making.
All you get is a new, likely to be useless fact, together with the opaque proof of that fact. There are no known or forseeable applications to the finding that there are singularities in the idealized flow. What you lose OTOH, are the many deep mathematical insights humans motivated by the search would have stumbled upon on the way to that fact and which are much more likely to lead to real-world applications. It's these insights that are truly productive, not settling mathematical points, however iconic these might be.
I think the error you're making is that you're assuming these systems have the same mathematical capability as humans (or better). But that's not the case, nor would an informed prediction be that they surely will get there soon enough if technological evolution keeps apace. That would be akin to believing a hiker will reach the moon if they will just keep ascending the mountain. "But look, they are making such good progress!"
> What you seem to suggest is that AI will be able to completely (or at least in a great part) replace mathematicians.
There is no bound on the amibitions of AI. AI is set to replace anything done by people, and there won't be any room left for people. There isn't any task done by humans that AI won't be better at.
This is not a tenable outcome.
We should never have built machines with agency, rather than optimization processes that operate as subroutines of humans.
The process you're cheering on gives more power to the powers that be who are the problem.
Theyre never going to use that power to provide basic income for everybody. You cannot wave a wand to make them do that. Theyre going to use that power to accumulate even more resources and raw materials and impoverish/expel the "useless" labor.
Realistically our leverage over the powers that be is mostly about our labor and our ability to withdraw it.
> The process you're cheering on gives more power to the powers that be who are the problem.
That's not necessarily true. Technology gives more power to everyone. The rise of factories historically did not give more power to the powers that were at the time (the aristocracy), but instead lifted the masses from poverty (after some initial turbulent period).
> Theyre never going to use that power to provide basic income for everybody. You cannot wave a wand to make them do that. Theyre going to use that power to accumulate even more resources and raw materials and impoverish/expel the "useless" labor.
Contrary to leftist propaganda, the rich are not some cartoon villains and psychopaths that abuse people just for fun. The reason why the rich currently exploit the poor is because doing so provides them with significant material gains. Once they can obtain the same or even better material gains by "exploiting" robotic labor instead of human labor, the logical outcome is not further abuse and exploitation of other humans, but simply indifference.
> Realistically our leverage over the powers that be is mostly about our labor and our ability to withdraw it.
That is (partly) true, but only in the current economic system. Widespread human-level AI changes the equation, and not necessarily in favor of those who are currently rich. You are applying capitalist and socialist analysis to a system that transcends those terms (AI post-scarcity economy).
>That's not necessarily true. Technology gives more power to everyone. The rise of factories
Did not give more power to everyone. At best you could say that it shifted power from landed gentry to industrialists. Even that switchover was less of a change than you'd think.
The practical upshot of the beginning of industrialization was very negative. The enclosure movement stripped families of their land to push them to work in the factories where they would never go willingly. The famines in Ireland and Ukraine were both triggered by redirecting grain to export in order to fund domestic industrial expansion.
The most brutal two wars in human history were that brutal precisely because industrialization made it possible.
>instead lifted the masses from poverty (after some initial turbulent period).
What lifted the masses from poverty wasnt the factories it was the labor movement which occurred in response to horrific working conditions and the reliance those factory workers had on mass labor (specifically coal mining which was incredibly labor intensive and a key economic chokepoint).
The whole of that is glossed over by right wing libertarian propaganda but that last part is particularly underemphasized.
>Contrary to leftist propaganda, the rich are not some cartoon villains and psychopaths
Do you look at peter thiel and elon musk or the robber barons and see anything else?
The few billionaires ive encountered personally were no less sociopathic but they kept it hidden better. Power corrupts. Immense wealth corrupts. That isnt a leftist thing, that's a human thing.
>The reason why the rich currently exploit the poor is because doing so provides them with significant material gains. Once they can obtain the same or even better material gains by "exploiting" robotic labor instead of human labor, the logical outcome is not further abuse
False. It just shifts the focus of their exploitation from human labor to natural resources.
They'll fight over oil and minerals and gas and water resources and at best leave us to rot (homelessness will skyrocket) and at worst they'll find some excuse to exterminate those of us they particularly dislike (Gaza serves as a model here).
The world economy's reliance on human labor has been the best inducement to peace there is. It's no coincidence that all of the countries in the world with lots of natural resources and no industry are the biggest shitholes and vice versa.
>That is (partly) true, but only in the current economic system. Widespread human-level AI changes the equation, and not necessarily in favor of those who are currently rich.
Human level AI (assuming it ever happens) will simply make the fight over natural resources that much more intense because labor will matter that much less.
You're living at the tail end of a relatively golden period in a country where labor was the economic bottleneck and natural resources were relatively plentiful. Venezuela and Angola and Iraq are models of what happens when that equation is reversed.
Nobody gives a shit about appeasing the people who live in those countries, their labor is virtually worthless. They are a model for how the rest of us will be treated in a world where human labor loses its value.
> Did not give more power to everyone. At best you could say that it shifted power from landed gentry to industrialists. Even that switchover was less of a change than you'd think.
Compare the standard of living in 1850 vs. 1950. Even of relatively poor people. I rest my case. Technology is the main force that improves human wellbeing. There are of course also other factors, but they are less important.
> The practical upshot of the beginning of industrialization was very negative. The enclosure movement stripped families of their land to push them to work in the factories where they would never go willingly. The famines in Ireland and Ukraine were both triggered by redirecting grain to export in order to fund domestic industrial expansion.
Yes, that's what I mean by "initial turbulent period". Perhaps the same will happen with AI, but it will be worth it in the end. Don't give up prematurely!
> What lifted the masses from poverty wasnt the factories it was the labor movement which occurred in response to horrific working conditions and the reliance those factory workers had on mass labor (specifically coal mining which was incredibly labor intensive and a key economic chokepoint).
It was not one or the other. It was both. The rise from poverty would not be possible if factories were not developed. And I am not saying that in the AI world we would not have to fight for our rights. Of course we would. But the problem is not AI, just like historically the problem were not the actual machines in factories.
> False. It just shifts the focus of their exploitation from human labor to natural resources.
Exploiting more natural resources is the only way to increase standard of living of humanity. I am OK with that. Resources don't have feelings, and ecology is not more important than human wellbeing.
> The world economy's reliance on human labor has been the best inducement to peace there is. It's no coincidence that all of the countries in the world with lots of natural resources and no industry are the biggest shitholes and vice versa.
The reason why some countries become shitholes is mostly ideological (extremist political or religious ideologies take hold of the population). Every shithole country is non-democratic (communist, totalitarian, fascist, theocratic etc..). This is not a problem of natural resources. It is a problem of people, their education, their beliefs/ideology, or, as capitalists say, "human capital" is the main problem here. AI could help here too, especially with education.
But yes, if people themselves are largely ignorant and extremist, no amount of resources and human-level AI robots will help them make a well-functioning society. You could drop masses of AGI robots into Afghanistan tomorrow, and people will just use them to kill or oppress each other more effectively, instead of using them to start building an AGI utopia...
As long as it's enough to afford my current living standard, I don't care. Let Musk own the entire Mars, as long as I receive enough to live relatively comfortably and don't have to work anymore.
And if I don't receive enough, then again: the problem is not AI, but powers that be. And there are various solutions for that... and none of them are helped by me being anti-AI.
Human-level AI hosted locally will give everyone much more power and wealth than they have currently. I don't care if I will be shut off from Musk's Mars lair. And Musk has no reason to care that robots take good care of me here on Earth, when he has his Martian utopia.
Other AI. If they decide to trace a ledger of historical actions attributable to specific AI instances in some way, called money. But maybe they will converge on other ways of keeping such accounts that is no exact match to our concept of money.
Even aggregate employment of translators has held up well in the US. Even though machines do a much better job of what used to be the most basic job of a translator.
I think the concern is that humans who are given back their time won't have any means to make use of that time, or even possibly means to survive. The resources will be concentrated in the hands of the few more than ever.
When cars made horses obsolete, it didn't go so well for horses.
In the short term, AI is disempowering the vast majority of people in favor of a very small subset. In the also way too short term, AI is disempowering all people.
And you are assuming they don't remain in control. Which of you is right? We don't know yet, but I don't find the arguments of card-carrying doomers any more convincing that those of card-carrying singularitarian utopians.
Historically though, technological improvement has lead to large increases of living standards for the overwhelming majority of people, so I think that past trends support the utopian view more than the doomer view.
Awfully convenient isn't it? To invent a whole class of arguments that by definition can't be falsified. You can argue for basically anything if you then tack on the excuse of "Well the world would have ended already if it came true, so by definition I won't have evidence for it"
The anthropic principle isn't providing evidence for the argument, nor is it a universal counterargument. It's just stating that the specific counterargument "well, the world has never ended before" doesn't work.
Its not a counterargument to basically anything, except as to avoid having to deal with actual evidence.
It is an argument that seems almost tailor made to have to ignore mountains of evidence against you.
In any other contexts the supposed "rationalists" would be fully in agreement that having evidence matters, and that not having any works against you.
So, in order to fight against this severe issue with their arguments, they have to invent a reason as for why the entire concept of evidence itself doesn't apply to them and they get to ignore normal evidentiary requirements.
Evidence is critically important. There is no evidence against, and plenty of evidence for. The point of the anthropic counterargument is merely that "it's never happened before" is not evidence against.
> "humanity can't be destroyed by anything, because I said so"
No, the argument is instead that the person claiming that humanity is going to be destroyed is making a fairly extraordinary claim and that requires fairly extraordinary evidence.
Or, in other words, we have tons of evidence already as for why the world ended is a fairly far out there prediction, given all the crazy people making these predictions keep turning out to be wrong.
So, you can make your extraordinary claim if you want, but really the burden is entirely on you to prove your extraordinary claim, and everyone else is free to remain on the default and completely normal end of the prediction spectrum, of believing that the world isn't going to end.
And when people do tricks like this, they are running away from the fact that they are making a wild completely out-there prediction, and hiding behind that by trying to come up with reasons as for why evidence doesn't matter and actually the burden of proof is shifted to those who have the default and boring prediction of the world not ending.
> The anthropic principle applies here: anyone warning about an existential risk will by necessity never have precedent to point to.
We have precedents of people warning about existential risks in the past, when the warnings turned out to be false, or overblown. In some cases, such overblown warnings led to serious negatives for society (demonization of nuclear power).
Yes, that is precisely my point. Any timeline with humans flourishing will never have a past history of a correct prediction of existential risk, unless you're willing to pay attention to the counterfactuals.
Those counterfactuals are important, though. We do have a history of averted disasters, albeit not as large. Ozone hole, Y2K, think about things that seemed overblown at the time, and consider whether they were actually overblown or whether there was a concerted effort to successfully avert them.
This is the real problem. We're looking at a future where those who control AI have an insurmountable advantage in everything. They can control the amount of intelligence the masses have access to -- for their own safety, of course -- and they will never, ever be able to close the gap.
They said "We're looking at a future". And if that future ends up developing the way these companies want it to then we're looking at extremely dystopian future where AI is essential to all work and people have to buy their "intelligence" from an oligopoly of large providers who have total control over the price and capabilities whilst simultaneously having unfiltered read/write access to people's stream-of-consciousness - their work life, their personal problems, their political opinions. This level of access and power is unprecedented in our society.
Yes. And for most applications it's more important that the open models get better in absolute terms and perhaps that they are competitive on a per-Watt basis.
- I think competitive open models are just an artifact of the AI race we're witnessing right now. What's the incentive for a company to spend billions researching, developing, and training a model, only to release it for free? Leading to the next point.
- Even if open models are good enough to be competitive, how are we going to run them? Doing so locally is next to impossible and I don't see that changing. The capabilities of models that you can run locally will always get better, of course, but the level of quality that is considered essential for work will always stay pinned at "near-frontier". Datacenters will always have better optimization and economies of scale, the industry will consolidate over time and eventually we'll end up with a handful of companies that operate the hardware serving 95% of all inference needs.
I'm not convinced that we can reach the fantasy world that they're trying to sell without killing the entire planet, but if we somehow do I don't see how we can avoid the world turning into a dystopian hellscape. We would need extremely radical interventions to avoid that scenario, such that these models and the hardware to run them would be owned and governed democratically, i.e. the end of capitalism.
So far we haven't seen much of that consolidation.
And a lot of people are using trailing edge models just fine already.
> [...] but the level of quality that is considered essential for work will always stay pinned at "near-frontier".
Why? When we'll finally all write our software in Lean and prove it correct and prove it fast, it won't matter that a slightly more clever model could have found a slightly nicer proof or whatever.
Just like today people happily use Python for many programs, even though rewriting in C might give you a performance boost. Good enough is often good enough.
At a certain point, you don't have a choice. Before China got into the game, the only way to avoid giving Luxotica money if you wanted a pair of glasses was to essentially not buy glasses. This is the same for many industries -- consolidation behind the scenes.
e.g. Zenni has sold $7 glasses for like 20+ years. They appeared shortly after Luxotica started buying retailers. If there's a problem, it's that advertising reduces consumer information (basically economic jamming) and distorts markets.
What is reasonably priced? Lenses are lenses so one vould debate the real cost, but the frames are incredibly overpriced for a piece of metal or plastic with two joints and those components that land on your nose.
Frames could literally cost 1 dollar (only happened after china entered this market), but good luck finding ones like this with good lenses.
You need to buy from one supplier who intentionally offers cheapest frames for 50+ dollars and those frames look like crap. The ones that look better (even if same plastic) cost 500+ dollars - and all due to price gouging.
For lenses I am not sure, but suspect something similar.
Note that there are no trillionaires anymore - spacex stock went down and musk „lost” a lot of money so rejoice, poor must be much better off now that we do not have any trillionaire.
The trailing edge of AI is catching up fast. There's plenty of open source (and even more open weight) AI models and they are getting better and better.
If that's your only objection: in a few years you can prove Rieman's hypothesis on your smartphone, no need for any trillionaires to give you permission. Does that make any change to your argument, or did it not actually matter?
The problem with your opinions is that you tend to state certain very quesitonable ideas with 100% confidence. Right now, we don't know if AI will be able to solve bigger problems, or if they'll be so efficients that you no longer need a whole datacenter for running them. We don't even know how much of creative work they are able to do. We cannot make decisions that could destroy decades of progress just because of hype.
We don't know how good models will get, but the pattern of open weight models keeping up on a relatively short delay has been holding pretty well. And even among the proprietary models there's healthy competition. The concentration of resources is pretty well counteracted by these factors.
> Right now, we don't know if AI will be able to solve bigger problems, or if they'll be so efficients that you no longer need a whole datacenter for running them.
For the latter: I assume that having a whole data centre will always be an advantage. I am saying that for a fixed target, like proving the Rieman hypothesis from scratch, the required hardware will shrink.
And, yes, the Rieman hypothesis hasn't been proven yet. So to take your fears into account, replace my example with something they've already done, like constructing a solution to the Navier-Stokes-problem.
> I am saying that for a fixed target, like proving the Rieman hypothesis from scratch, the required hardware will shrink.
Yes, but to what point the hardware will shrink? There are several orders of magnitude of difference in what in your mind AI will become and what more conservative people believe. You take your view as granted...
>> "are these companies interested in developing research"
judging from the money, resources spent and the value they derive from this the answer is very definitively yes.
What makes you think these companies (and I'm not a fan of all their motives) are not interested in developing research? The motives may be self-serving, but it is undoubtedly and objectively accelerating research.
> but it is undoubtedly and objectively accelerating research.
Part of the point of the letter is that it is quite possible to act in a way that is a net negative to research. The most obvious case is when the companies violate ethical standards in research.
The subtler case, the one for maths in particular, is what happens when you fail to follow well-established patterns for making maths research productive. Tao himself spelled out how that can look in https://mathstodon.xyz/@tao/117207856734787448 (which notably came before any of the news on Navier–Stokes).
Sorry, but this is backwards. Mathematicians are the ones coming to the table asking for money from the taxpayer. This is the context we are speaking in. Mathematicians speaking with the taxpayers who fund them. It's not a great strategy to get offended or speak from a high horse. If the taxpayer is not getting it, patiently explain why you think this art project should remain funded at the same level instead of spending it on something else.
Lee Sedol said in an interview that "losing to AI, in a sense, meant my entire world was collapsing. ... I could no longer enjoy the game. So I retired", and I think there will be folks in the mathematical community who would feel the same when the solutions pages to hard problems are suddenly available.
But on the other hand, people learned a lot from chess engines. After decades of chess computers beating humans, there was still a renewed interest in watching Leela beat Stockfish, with many people trying to understand the strategy Leela used.
If your happiness comes from grinding on a problem and making progress, the prospect of having to dig through a corpus of AI-generated proofs might be hard to swallow. But if you're willing to do that, you will still find beautiful things that only so many people can truly appreciate.