So what is the deal with their agenda here? Understanding that Lesswrongers are a conservative group with some radically authoritarian ideas
The popularity of libertarianism with them also seems distinctly in favor of power hierarchies rather than the libertarian communist types. So when these viewpoints stoke fear I wonder what what kinds of solutions they are driving at
I think you're looking at this through the wrong lens. There are a variety of issues that the Less Wrong community gets ironically wrong.
But the agenda here is very simple: make sure AI doesn't kill us.
They believe that earnestly. That is the end goal. Everything is in service of that not the other way around. And with how quickly AI is progressing I'm pretty onboard with that.
"The agenda of Christianity here is very simple: make sure the devil doesn't kill us. Christians believe that earnestly. That's the end goal. Everything is in service of that and not the other way around. And with how quickly the world is turning to shit, I'm pretty onboard with that."
There's a blind devotion to a group's chosen x-risk event, alongside assigning infinite utility to that one goal at the exclusion of all other useful goals. That's an important hallmark of fundamentalism. I sincerely don't think it's a good thing.
Except gods and devils have never and will never exist but AI might. Although I think it incredibly unlikely that AI will ever be in a position to create a human exterminating Skynet totality and the discussions should be more about how humanity is affected by machine learning, as it has been by previous technological advances. The LessWrong community try to address bias but seem quite biased here by sci-fi writers.
Christians genuinely have the faith that gods and devils exist. It's a genuine belief, they're convinced by whatever evidence they see, even though outsiders don't necessarily understand. Maybe they're not 100% there, but that's okay - they plan for this low probability but important outcome by going to church, helping their neighbor, performing outreach, converting nonbelievers, discriminating against queers and bitching about homeless people, pretending to read bible, etc.
You have faith that AGI will both exist and have similar power. That's your genuine belief, you've been convinced by the evidence that you see. Maybe you're not 100% there but that's okay - you plan for this low-probability but important event by reading up on machine learning, contributing to your community discussions, studying alignment, performing outreach, converting folks who don't believe in AGI, pretending to read Norvig textbook, etc.
Your analogy keeps getting more strained, IMO. Should we believe that unicorns and dark matter are equally plausible? Neither has been directly observed, after all. I'm not sure this line of argument is getting you anywhere. It would be more productive, I think, to directly engage with the argument and say why you think concern about AGI is implausible.
> You have faith that X. That's your genuine belief, you've been convinced by the evidence that you see. Maybe you're not 100% there but that's okay - you plan for this low-probability but important event by doing Y.
Is there any low-probability but important event that doesn't follow this template?
> one goal at the exclusion of all other useful goals.
I don't believe this to be true. Nor do I believe the AI research community broadly believes this. Other goals can certainly coexist.
But when the median AI researcher believes there's a 10% (!) chance that what they're working on will destroy the world, and the pace of that work is only quickening, that seems like a big issue to worry about.
> But when the median AI researcher believes there's a 10% (!) chance that what they're working on will destroy the world, and the pace of that work is only quickening, that seems like a big issue to worry about.
Yet they will not quit their job. What that suggests to me is that they don't really believe it, otherwise they would sacrifice their paycheck to stop it. Seems to me that what they really believe is that they should get more attention and more funding.
> Yet they will not quit their job. What that suggests to me is that they don't really believe it, otherwise they would sacrifice their paycheck to stop it.
I've talked to quite a few researchers. They generally do earnestly believe that AI has huge damage potential. The answers to why they keep developing AI falls in several different buckets (to which I have varying degrees of agreement)
1. We're in a prisoner's dilemma. If everyone could unilaterally agree to slow down AI research I'd be happy to quit, but until that happens if I don't do it someone else will and with even fewer scruples.
2. The only way to develop AGI safely is to push things to the brink. It's impossible to make progress on how to make sure that AGI is safe is to almost make AGI because not until then do you have a clear enough picture of the exact form AGI will take to do productive safety research.
3. Slowing down AI research has its own risks. If you slow down AI research, but other fields of technology continue to progress, you can end up in a scenario where everyone's personal computer is capable not only of running models, but of doing cutting-edge ML training. Right now to coordinate on developing AGI safely, there are maybe ~10 top teams with the infrastructure possible to do the training (not just running the model!) needed to advance the state of the art you have to get cooperation among. Hard, but not impossible. If technology progresses to the point where there tens of thousands of teams all capable of meaningfully advancing the state of the art then coordination is absolutely impossible.
4. If AI can, then AI should replace humans entirely. (the most chilling response I've heard. Luckily few in number, but it shows up even on HN)
That's a very strange analogy. What about Christian doctrine includes "making sure the devil doesn't kill us"? "Making sure the devil doesn't tempt us into evil" maybe? But Christianity kind of assumes we will fall into temptation and be forgiven anyway.
As other commenters point out, the end is not the same as the means. Simple agendas, when followed with sufficient vigor, can sometimes cause enormous amounts of harm. I’ve recently encountered a number of posts from this community that are filled with despair about the speed at which AI is progressing, and predict the end of humanity on a short timeline. Some are written by very well-known figures in the community. Other adjacent figures have mooted the idea that a police state might be the only way to control certain technologies. It’s reasonable to ask whether existential terror might lead some in this community to make rash decisions. It’s also very concerning because these existential fears can distract people from thinking about other, but perhaps more relevant concerns. Imagine that AI doesn’t resist human orders, but the people giving the orders use it to radically restructure our society in ways that make our current economic system incompatible with human flourishing: is this community concerned about that (perhaps very realistic) outcome, or do their interests end when AI is compliant with its masters?
> is this community concerned about that (perhaps very realistic) outcome, or do their interests end when AI is compliant with its masters?
The short of it is yes. As far as I'm aware the long-term hope in the Less Wrong community or more generally the AI safety community is that AI becomes available for everyone and under the control of everyone.
But the first, paramount step is to make sure it doesn't destroy everything and recognize that such a guarantee doesn't come for free, but rather requires an enormous amount of work and coordination.
my original question was what realm of solutions are they advocating
I don't take lightly to suggestions that we enact police states, highly authoritarian government controls, and other state or capital extremes that I've seen that community / the alignment folks advocate for as you can find when you look past their concern and fears
> my original question was what realm of solutions are they advocating
They don't really have an answer, which is why they're freaking out.
> I don't take lightly to suggestions that we enact police states, highly authoritarian government controls, and other state or capital extremes that I've seen that community / the alignment folks advocate for as you can find when you look past their concern and fears
What comments are you thinking of? And what solution would you advocate for in a scenario where creating superhuman AGI that kills everyone is much easier (and probably the default) than creating superhuman AGI that doesn't (basically how do you avoid prisoner's dilemma-esque scenarios given how much power AGI gives its makers right before it kills them)?
In my experience, they do tend to have a remarkably authoritarian slant. Only in their view the people running everything and doing all the gatekeeping should be rational, intelligent people (like them, natch), not just whoever has the power to actually get these things done.
We have words for this already: technocracy, authoritarianism, etc.
the liberals among them or capitalist libertarians don't round out the spectrum of political views when it comes to ideas that community popularly advocates.
advocates for strong gatekeeping and violence against any public subversion of it, within a centralized power whether capital or state, is getting at different flavors of the same kind of tyranny that you'll end up with whether liberal, conservative, capitalist libertarian.
Well, they definitely do not identify as authoritarian according to the annual surveys. Very few claim to be conservative or reactionist, most are either libertarians, liberals or social democrats.
Anyway, the rationalist community was never about people who were particularly rational, it was all about people recognizing their own irrationality and biases that inescapably comes with being equipped with a human mind. Of course, that does not mean it didn't attract people who thought it was about being a tribe of us, the rational and enlightened people vs them, the irrational plebs. Which is rather ironic, of course.
The only way I could parse it was the following logical chain: Lesswrongers definitely bend more libertarian than the rest of the population, and the common aphorism that "libertarians are just conservatives who like to smoke weed".
Although, with a few conversations that I have had with the "alignment" enthusiasts gave me the impression that in their opinion, many of the hot button issues like climate change/social inequality/global poverty etc. are irrelevant and not worth spending time over, because AI alignment is a much more serious threat. In that view, 90% or more of their political position overlaps with the conservative crowd ("issue X/Y/Z is not worth our time, and we should just let it happen as it does now"). So I am not surprised an outsider may label them as conservative.
There are strong parallels between a belief in AI supremacy (roko's basilisk and the like) and belief in divine punishment (pascal's wager and the like). I've heard people mockingly describe superintelligence as "catholicism for nerds," so I could certainly entertain the idea that lesswrongers are conservative.
If i believed in socially progressive lib values, but thought a non-explicitly progressive issue was a major immanent threat... you'd consider me conservative for the structure of my priorities?
What specifically gives you that impression? They describe their experience talking to alignment people, characterise it with the 90% rule, and describe their preference for tackling the 90% issue as in line with apathetic or ideologically opposed to remedy conservatives. They then say they are not surprised outsiders might label them conservative - _just as they have_.
They are describing their own view _and_ attributing their reasoning to others.
Nope, they're just saying that some people worrying about hard AGI takeoff argue that alignment is a more important problem to solve than many others because a) an aligned AGI can help us solve all the other problems, and b) an unaligned AGI can make all the other problems irrelevant due to, well, human extinction.
So if you have liberal values and consider a hard takeoff a likely near-to-medium-term scenario, then it's perfectly reasonable to argue that you should prioritize alignment research over almost anything else, because it's a critical instrumental goal for achieving all your other goals. This by no means makes you any less of a liberal, or more of a conservative.
But people who are more interested in virtue signalling than results, and/or don't know about or believe in a hard takeoff scenario, may interpret this train of thought as "conservative" because it appears to not match the standard thought patterns that a good, card-carrying liberal is supposed to have.
Just because people decide to focus their time on one issue (say AI), doesn't mean that they don't appreciate people working on other issues. Another example: I can appreciate a local soccer coach without having to believe that their work is equally important as that of the president/prime minister.
The last I saw (~10 years ago), the small but vocal neoreactionary minority (countable with the fingers of a single hand) of the lesswrong crowd went and founded their own blog (More Right, now defunct) and everybody else were happy to see them go if it meant less spamming Less Wrong with cringe stuff like race realism and pickup artistry.
In all LW surveys to date, people who have identified as "conservative" or "reactionary" have been a tiny minority. There are many libertarians (whether "l" or "L" or whatever libertarianism means to different people), but the "liberal/social democrat" block is the largest. With regard to American party affiliation, the large majority lean Democrat. Most are atheist and lean consequentialist ethics, neither being a sign of conservative or reactionist values.
because there's a difference between self-reporting on one hand, and observed behaviour and the confessions of those involved when they don't think anyone's watching on the other, e.g. https://archive.is/LA95f
