Words! What are they for? Communication. The word "want" is clearly used here not as anthropomorphism, but as an accurate descriptive term of the output of a black-box system, whether the internals are human or otherwise. They've elaborated on how the usage is intended elsewhere.
> Once AIs get sufficiently smart, they'll start acting like they have preferences—like they want things. We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way.
> If you play chess against Stockfish—the best chess AI at time of writing —it won’t squander its queen. Does Stockfish “want” to defend its queen? Does it “want” to win the chess game?
> That’s between you and your dictionary. As for how we use the word in this book, when an AI like Stockfish defends its pieces, lays traps, takes advantage of openings in your defenses, and winds up winning, we’ll describe it as “wanting” to win. In saying this, we’re not commenting one way or the other on whether a machine has feelings. Rather, we need some word to describe the outward winning behavior, and “want” seems closest.
(From the book "if Anyone Builds It, Everyone Dies" by the same authors, so I assume it represents their position)
> Their thesis may well be right but this example is completely wrong
And then I explain in detail how it is wrong.
The words you quote (notably, the word "want") are not in the statement from TFA that I addressed, and I'm not about to address this goalpost moving here. I do understand (especially from Daniel Dennett's Intentional Stance) how AI agents can be viewed as intentional systems, but the errors in the text I quoted are of a different sort. And again,
> All that said, there ARE "relentless" systems with goal-driven behavior
But a program that does no more than always play the move with the highest Stockfish eval isn't one, any more than a program playing "pick the higher number" that always selects your last move + 1. "It's diabolical! It's relentless! It wants to win and it doesn't care about me!"
If they want to drive home their point then they need more compelling examples.
> Once AIs get sufficiently smart, they'll start acting like they have preferences—like they want things. We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way.
> If you play chess against Stockfish—the best chess AI at time of writing —it won’t squander its queen. Does Stockfish “want” to defend its queen? Does it “want” to win the chess game?
> That’s between you and your dictionary. As for how we use the word in this book, when an AI like Stockfish defends its pieces, lays traps, takes advantage of openings in your defenses, and winds up winning, we’ll describe it as “wanting” to win. In saying this, we’re not commenting one way or the other on whether a machine has feelings. Rather, we need some word to describe the outward winning behavior, and “want” seems closest.
(From the book "if Anyone Builds It, Everyone Dies" by the same authors, so I assume it represents their position)