My background is in cognitive science and psycholinguistics. I spent more than ten years talking to first year psychology undergraduates about whether AIs could be conscious; also did some research on (extremely tiny) AIs in modelling language behaviours.
There is a great deal of good thinking on Chiang's topic by professional philosophers, and there's much to be said for reading them. I won't rehearse their ideas here. Chiang's arguments might be correct; but I suspect they probably aren't, and his error may well stem from characterising human thought as something in its own class, which is probably a cognitive bias that humans have. He might also - I'm speculating - be arriving at his conclusion based on his feelings, which the final paragraph suggests (the comment about the models being based on morally dubious actions).
Speculation aside, we are not, I believe, in a position to make points like he does with any certainty.
I have a similar long term interest in this field.
It has been quite frustrating encountering arguments that have been extensively debated for years be presented as if they were new revelations.
In all my debates with people in the last few years I have primarily taken the position of trying to explain the problems with claims of certainty, and that lack of certainty permits possibility of the opposite. We should act responsibly around what might be possible.
There is also the narrative of "being too obsessed if you could to consider if you should" or similar claims of an unconsidered path forward.
Isaac Asimov wrote the first of the Robot stories in 1940, they were not written in isolation, it came from an awareness of the situation and the questions that must be asked. There was a community considering these issues. Asimov gave the wider public a view of some of those issues.
If we have a hundred years of people going "This is coming, we had better decide what we want it to be" and nobody listens to them, or frequently outright ridicules the need for considering their ideas, why is it now we are placing the blame on those who are now showing some success at what they told us they were attempting all along.
On Asimov, much as I loved those stories as a teenager, it was only later that I realised the robot stories are largely explorations of how apparently sensible rule-based systems generate unexpected and sometimes harmful outcomes. The Three Laws are presented as hard guardrails, but the stories focus on ambiguities, loopholes, and unintended consequences. I'm not sure how intentional this was; he might have been attempting to make a point about rule-based systems, or perhaps he was following his instinct for drama.
The books were absolutely an exploration of the suitability of the rules, combined with asking what do we actually want.
He was very clear to stipulate that the laws themselves were more than the text that represents them, no word play or creative interpretation was possible. This is much easier to do as an author than it is as a developer. You just declare that part as having been susessfully done by scientists.
The rules were their clear semantic meaning. Some of the stories explore the implications of changes to the laws either by design or accident.
I feel like the most interesting ones are the stories where the laws are working and have undesirable outcomes. It reveals that even if fully obeyed, they do not represent what we want.
Some of that is because what we want is dependent on situations that can go beyond safety and harm. Some of it is outright human hypocrisy.
When he gets to things like a closeted robot running for president he makes thebpoin clear about how if the bodies appear alike, you can't tell the difference by the behaviour of a robot conforming to the laws and a good person. That led to the obvious way to distinguish them was to get it to do something bad. The solution on how to fake that might have been a subtle dig at class based society.
It's been some 20 years since I last read the stories, I wonder if there's any point now given my ability to record new information in my youth significantly exceeded my ability to retain it now. I suspect after a week my youthful recall would be better than my week old recall.
His certainty seems like a rhetorical style rather than a series of facts. It would be very annoying and not persuasive if he started every paragraph with some variation of, "I think" or "I believe."
I like that he makes the emotional component of his argument plain. I'm deeply suspicious of anyone would try to argue about the concept of personhood and consciousness using only logic and empirical "correctness."
My background is in computer science and linguistics. I wholeheartedly agree with you: We humans are dubiously-equipped to determine whether or not AI could be conscious.
I'm also super curious to learn more about the philosophers you referenced and their thoughts on this subject. Would you be willing to share some of your favorite examples?
I could spend a very long time double-checking my sources, but I want to reply tonight, so please take the following with a pinch of salt, since it's largely from memory:
0. Descartes/Titchener/Chomsky and friends for background.
1. John Searle featured prominently because of his accessibility. I tended to present his Chinese Room argument as a criticism of symbolic AI alone, though I believe he considered it a criticism of sub-symbolic approaches as well.
2. Thomas Nagel's classic article "What Is It Like to Be a Bat?" is a good introduction to qualia, which is how we describe direct conscious experience.
3. Wittgenstein would be important in terms of the impossibility of empathising with, or seeing things from the perspective of, other minds that have emerged in different contexts (such as animals). However, I rarely spoke about him because, frankly, I couldn't understand him in the original!
4. David Chalmers. Writes clearly (and coined?) the 'hard problem' of consciousness and why subjective experience appears difficult to reconcile with a purely physical account of the mind.
4.1 Daniel Dennett. The clearest and most influential critic of the idea that consciousness presents a special explanatory problem.
5. Darwin and others. Comparative psychology (the study of animal minds) strongly suggests, by showing that the antecedents of the human mind are present in animals, that we should reduce our bias that human minds are special, ineffable, or somehow atomic.
6. Jerry Fodor. Modularity: the idea that the mind is composed of modules that specialise in certain tasks (e.g. phoneme perception, syntactic analysis, face recognition) and operate largely unconsciously unless they are dependent on one another in some way. This helps us take a computational approach to the mind. It reminds us that much of what we do mentally might be expressible in computational terms.
When (not all) neurons die off and are replaced by new ones doing the same job, our sense of self and identity is somewhat cast into doubt. In a physical system, if consciousness is a magic or non-material entity of some kind, what is happening to it during this process?
2. Integrated Information Theory. A good attempt to tackle consciousness rationally. The idea is that consciousness corresponds to, or is associated with, the degree to which information in a system is integrated into a unified whole.
Is there an accessible way for a layman such as myself to read about some of these ideas (Really I mean philosophical discussion in general) without having to read entire books? Is there an active HN-equivalent or wiki or something?
I haven’t come across one, unfortunately. You’ll forgive the irony if I suggest talking to a frontier model like Claude about these issues; they are quite accurate, although they’ve been seeded with biases that make them lean in Chiang’s direction. Otherwise, the Stanford Encyclopedia of Philosophy is excellent as a starting point:
(philosophy and computer science background, but that's long in the past I just do engineering for big corpo)
They're not exactly casually absorbed, as in a wiki or forum. But you can read some books that begin to introduce these ideas. On the topic of consciousness, less academic and more slated towards general audience: Reality+ by David Chalmers and Mind and Cosmos by Thomas Nagel and Galileo's Error by Phillip Goff will give you and interesting gamut of ideas.
The thing about arguments in philosophy is that they span from a very old web of thought that has been refined into very sharp positions over a long time. So you will find yourself ever recursively going back to understand ideas and framing with more precision.
This is why it's difficult to casually get into these topics IMO. There's just been so much said and discussed, to understand the current meta (as the gaming folks might say), you have to understand how we arrived at the current meta. And that's a long journey that's never complete!
His book 'On Liberty' is the subject of a recent In Our Time episode (BBC Radio Four series on the history of ideas) [https://www.bbc.co.uk/programmes/m002pqnc]. They discuss his childhood and his (apparently very warm) relationship with his father. (Sidenote: first proper In Our Time episode with the new host; he seems fine, but I miss Melvyn Bragg.)
I'm half way through this article. The word 'introspection' might be better replaced with 'prior internal state'. However, it's made me think about the qualities that human introspection might have; it seems ours might be more grounded in lived experience (thus autobiographical memory is activated), identity, and so on. We might need to wait for embodied AIs before these become a component of AI 'introspection'. Also: this reminds me of Penfield's work back in the day, where live human brains were electrically stimulated to produce intense reliving/recollection experiences. [https://en.wikipedia.org/wiki/Wilder_Penfield]
Regardless of some unknown quantum consciousness mechanism biological brains might have, one thing they do that current AIs don't is continuous retraining. Not sure how much of a leap it is but it feels like a lot.
Beyond consumer-producer relationships, there are many instances where an individual is required to deal with a baroque interface, as I just did when starting to look after an ill parent and figure out what care they could get from the local and state governments; there are forms, definitions to get one's head around, high stakes (get it wrong and you could be breaking the law), and so on. An AI in this case was incredibly helpful, particularly when I was overloaded cognitively and emotionally. There is no particular incentive on the other end of the citizen-government relationship for the government to obfuscate things, but things are sometimes very complicated and provided in verbose language. For those interactions, for that asymmetry, an AI will be very useful.
Indeed. But the unintended consequence (perhaps) of LLMs making things easier to use is that more people will use them - basically Jevons paradox.
I would expect that this will cause certain programs to see more demand than the creators anticipated for (extrapolating previous trends), which might require changes in the programs (i.e. more people apply for benefits than expected, benefits / application might have to be cut, etc).
And in some ways there's a Cantillon effect (though traditionally associated with proximity to the "money printer", but here the proximity is to the LLM-enablement; in that those who use the LLMs first can get the benefit before the rules are changed).
A quick note to say that at our local repair cafe we do a roaring trade in vacuum repairs for peanuts (not literal peanuts; though some of our repairers do get peckish). If you're in Europe, there's chance you have one nearby. https://repaircafe.org/en
Could you please go in to a bit more detail on how you set that up, how you handle issues, etc? I'm a member of a makerspace, and there's been discussions about doing something similar. There's just not a consensus about how to go about it.
...generally I've seen weekly/monthly "fix-it" workshops as a kindof open-house / membership drive.
Probably best to 1) have people sign safety waivers, especially if they're not members 2) have people sign a "we can offer to help you try, but your widget might end up worse than before" waiver 3) run it as a volunteer outreach event with a focus on getting membership rather than a transactional "fix my ____ for free" outcome
My summer project for my comp sci MSc was an open source natural deduction helper tool for the Mac, called Baker Street. I let it lapse simply because I didn't have time to look after it, but now it's available for download once more: https://apps.apple.com/gb/app/baker-street/id1528304157
My codebase contains a lot of Objective-C but I'm slowly replacing it with Swift. I rarely write Objective-C (largely because it's so easy to introduce bugs with release/retain shenanigans) unless I really have to. Interoperability between the languages does work but is fragile, with Xcode often getting into a chicken-and-egg state where a compilation error in Swift prevents the Objective-C from compiling, and vice versa, stuffing the interface with errors that can make it hard to discover where the problem is.
There is a great deal of good thinking on Chiang's topic by professional philosophers, and there's much to be said for reading them. I won't rehearse their ideas here. Chiang's arguments might be correct; but I suspect they probably aren't, and his error may well stem from characterising human thought as something in its own class, which is probably a cognitive bias that humans have. He might also - I'm speculating - be arriving at his conclusion based on his feelings, which the final paragraph suggests (the comment about the models being based on morally dubious actions).
Speculation aside, we are not, I believe, in a position to make points like he does with any certainty.
reply