> ... "it doesn't seem likely" aren't the strongest arguments.
Since we're talking about the future, it would be incorrect to talk in absolutes so speaking in probabilities and priors is appropriate.
> Our intuition isn't a good guide here.
I'm not just using intuition. I've done as extensive an evaluation of the technology, trends, predictions and, most importantly, history as I'm personally willing to do on this topic. Your post is an excellent summary of basically the precautionary principle approach but, as I'm sure you know, the precautionary principle can be over-applied to justify almost any level of response to almost any conceivable risk. If the argument construes the risk as probably existential, then almost any degree of draconian response could be justified. Hence my caution when the precautionary principle is invoked to argue for disruptive levels of response (and to be clear, you didn't).
So the question really comes down to which scenarios at which level of probability and then what levels of response those bell-curve probabilities justify. Since I put 'foom-like' scenarios at low probability (sub-5%) and truly existential risk at sub-1%, I don't find extreme prevention measures justified due to their significant costs, burdens and disruptions.
At the same time, I'm not arguing we shouldn't pay close attention as the technology develops while expending some reasonable level of resources on researching ways to detect, manage and mitigate possible serious AI risks, if and when they materialize. In particular, I find the current proposed legislative responses to regulate a still-nascent emerging technology to be ill-advised. It's still far too early and at this point I find such proposals by (mostly) grandstanding politicians and bureaucrats more akin to crafting potions to ward off an unseen bogeyman. They're as likely to hurt as to help while imposing substantial costs and burdens either way. I see the current AI giants embracing such proposals as simply them seeing these laws as an opportunity to raise the drawbridge behind themselves since they have the size and funds to comply while new startups don't - and those startups may be the most likely source of whatever 'solutions' we actually need to the problems which have yet to make themselves evident.
Since we're talking about the future, it would be incorrect to talk in absolutes so speaking in probabilities and priors is appropriate.
> Our intuition isn't a good guide here.
I'm not just using intuition. I've done as extensive an evaluation of the technology, trends, predictions and, most importantly, history as I'm personally willing to do on this topic. Your post is an excellent summary of basically the precautionary principle approach but, as I'm sure you know, the precautionary principle can be over-applied to justify almost any level of response to almost any conceivable risk. If the argument construes the risk as probably existential, then almost any degree of draconian response could be justified. Hence my caution when the precautionary principle is invoked to argue for disruptive levels of response (and to be clear, you didn't).
So the question really comes down to which scenarios at which level of probability and then what levels of response those bell-curve probabilities justify. Since I put 'foom-like' scenarios at low probability (sub-5%) and truly existential risk at sub-1%, I don't find extreme prevention measures justified due to their significant costs, burdens and disruptions.
At the same time, I'm not arguing we shouldn't pay close attention as the technology develops while expending some reasonable level of resources on researching ways to detect, manage and mitigate possible serious AI risks, if and when they materialize. In particular, I find the current proposed legislative responses to regulate a still-nascent emerging technology to be ill-advised. It's still far too early and at this point I find such proposals by (mostly) grandstanding politicians and bureaucrats more akin to crafting potions to ward off an unseen bogeyman. They're as likely to hurt as to help while imposing substantial costs and burdens either way. I see the current AI giants embracing such proposals as simply them seeing these laws as an opportunity to raise the drawbridge behind themselves since they have the size and funds to comply while new startups don't - and those startups may be the most likely source of whatever 'solutions' we actually need to the problems which have yet to make themselves evident.