Their concern isn't biased AI algorithms. Their concern (well, one of their main concerns) is that a super-intelligent AI could wipe out humanity sometime in the next 100 years. They have arguments that this could happen even if the AI wasn't malicious.
I'm a bit skeptical too (and I haven't looked into that much), but it's not obviously stupid. They'd point out that even if there's just a 1% chance of humans being wiped out in the not-so-distant future, that's a big deal (and a large expected value for number of deaths) and we should work on reducing those odds.
A lot of this rests on the assumption that super-intelligence is really powerful. Like, take-over-the-world type of powerful.
That's what I had in mind with my "tiny probability, infinite timescale" multiplier, and also why I said "waste of money". The comment I replied to had a more nuanced point about AI research, which I think merits a more nuanced approach.
Having said that, I repeat what I said at some point: once they show that they can stop the dumbest DDOS, then and only then I'll listen to what they have to say about a super-intelligent AI. If they can't do even that, then I don't know why I would listen to anything else they have to say.
No, what I mean is: some effective altruists invest their time and money in stopping a possible future AI from slaving humanity.
So, my challenge to them is: if you think that you can stop a super-intelligent AI from taking over, show me that you can stop the dumbest possible malicious intelligence that we know of, which right now would be a DDOS. And the incentives to develop either are pretty much the same, too.
If they can't, and my guess is that they can't, then I don't see why I should believe that they can stop anything bigger.
It's weird to demand AI safety researchers should stop working on AI safety and prove they can mitigate DDOS attacks. The two are nothing alike. A DDOS attack isn't an intelligence. DDOS protection services work largely by having more resources and infrastructure than the attackers. Anyone can do that with enough money.
First, I want to state that I appreciate your comment - I'm about to use some strong words in my reply and I wanted to say that I'm using them for conciseness, and not out of disrespect towards your (valid) points.
If someone were to argue "our strategy for survival is not to fight our enemies, but rather to convince them to use non-lethal weaponry", you would read their research in the rubble of their country after an enemy laid waste to them. I feel the same way towards that line of research: why would the US Army program their AI to be less aggressive when they could... not? How about the Russian army? China? Iran? You point out that anyone with money can make a DDOS attack, and I feel exactly the same way here: malicious AI could come up from anywhere.
If those researchers truly believe that a malicious AI is possible (and again, the website that started my comment chain puts it as a top-3 priority), they have to assume that it will be developed by someone with no interest in playing nice, just like that factory that started spewing ridiculous amounts of CFCs in 2019. Why would anyone use anti-bias correction in their NN embeddings when the biased ones exploiting harmful stereotypes gets them higher profits?
If those researchers cannot stop the most likely scenario, then I consider their research little more than wishful thinking. And we have seen how good "everyone will surely play nice" has worked - spam, pop-ups, phone scams, the list goes on. That's why I like DDOS as an application: it's the world's stupidest AI causing a lot of trouble. Can you outsmart that? Good, then now we can talk about outsmarting Skynet.
Okay, I see your point about hostile states, and I appreciate the respectful debate.
I'm not so sure about AI safety research myself, because it's very hard to evaluate how effective it is (I guess if an AI ever wipes us out, we'll have a data point), and because I don't know very much about AI safety research.
But let's suppose that super-intelligence is really powerful and dangerous (suppose there's a 10% chance it could kill all humans, despite our intentions). Now what? Is there anything more productive we could be doing to prevent that than just waiting?
Let's further suppose that the US and Chinese militaries want to make aggressive AIs that will subdue, or worse, exterminate their rivals.
As outlined in the AI Control Problem wikipedia article, there's a lot of concern that we'll make a super-intelligence that harms us completely by accident. For example, if you build one with the goal of protecting citizens, it might reason that to protect people, it must continue to exist. Therefore it undermines and outwits anyone who wants to shut it down. Worse, it might reason it could better protect people if it had more political power and more physical resources. Or even worse, if we really mess up how we programmed its goal, it might reason we're best protected if we're all put in a permanent coma and stored in a concrete bunker.
So even if the US and Chinese militaries were fully evil and wanted to exterminate all other countries, they might still want to use results from AI safety research, just to ensure they don't accidentally destroy themselves. And to some extent, for humanity's sake, making an AI with safety that exterminates every country except the one that created it is still slightly better than an unsafe AI that exterminates everyone.
I don't think the US or China or Russia would want to exterminate other countries. But whatever they choose to do with their AI, the AI safety research is there to ensure they don't lose control of their own AI. If anything, those safety features should be even more desirable if you're building an aggressive military AI.
Finally, how do you fight a hostile super-intelligence? I think I know what these AI researchers might say: you need to have an even smarter friendly super-intelligence. What's more, a friendly AI could look for and destroy other AIs while they're still in development. So maybe the key to avoiding hostile super-intelligences is just to be the first to make a super-intelligence and ensure it's safe and friendly. And if we ever get in an arms race to build the first super-intelligence, we better hope AI safety is well researched and understood by then. Because if not, someone may create an unsafe AI just to have it before their rivals.
I'm a bit skeptical too (and I haven't looked into that much), but it's not obviously stupid. They'd point out that even if there's just a 1% chance of humans being wiped out in the not-so-distant future, that's a big deal (and a large expected value for number of deaths) and we should work on reducing those odds.
A lot of this rests on the assumption that super-intelligence is really powerful. Like, take-over-the-world type of powerful.