What it's like - the gestalt of a bat (or other thing) as it engages its sensing-deciding-reacting loop. This gestalt isn't just for biological organisms, but any system for which its decision making engages with representations of the external environment unified with a self-representation to form a coherent representation of a persistent entity engaged with an external world.
Why do such systems need this gestalt? Why consciousness instead of everything happening in the dark? The recognition of oneself as situated in the world is crucial to coherent engagement with the world. It is how an entity can ensure its body parts are moving towards the same goal. It's how behavior over time doesn't undermine its purpose. Fragmented, incoherent behavior does not serve self-preservation.
LLMs as they are currently constructed probably aren't conscious, but we are a hop skip and a jump away from ones that are.
I agree that evolution could not produce a rational agent who would still reliably respond to lower level imperatives (such as pain, hunger, lust) without consciousness and feeling. The primitive parts of the brain have to be able to override the higher functions to ensure survival and reproduction. But an LLM isn't evolved in this way; its fitted to a functional output. It is entirely possible there will never be anyone home. I sure hope there isn't, because at the scale we're using them it would be a moral catastrophe.
> The recognition of oneself as situated in the world is crucial to coherent engagement with the world. It is how an entity can ensure its body parts are moving towards the same goal. It's how behavior over time doesn't undermine its purpose. Fragmented, incoherent behavior does not serve self-preservation.
Why would movement towards a goal be incoherent if it happened "in the dark"? Our brains perform many critical functions "in the dark" (and do so coherently) which do not rise to the level of consciousness.
Presumably the question you're asking is why does a unified self representation require consciousness. (Split brain cases are easy examples of how a break in unification results in incoherent behavior.) The brain nominally performs functions as cascading behavior of atoms whose structural relationships correspond to various functions. But there is no unification at the unconscious/atomistic level therefore a new representational regime is required that can ground the higher level unification.
A successful organism exhibits a high level of competence at reacting appropriately to environmental/sensory states. The "light's being on" is how the brain represents being situated in a world and the significant features therein. Representations within this gestalt are inherently meaningful. For example, phenomenal pain brings with it competence at protecting bodily integrity. The memory of pain becomes part of the explanatory narrative for the monitoring function that tracks progress towards goals ensuring coherent behavior (imagine being fearful of a stove but not knowing why). The contents of consciousness is the semantic engine that induces competent behavior over time on otherwise naive entities.
> For example, phenomenal pain brings with it competence at protecting bodily integrity. The memory of pain becomes part of the explanatory narrative for the monitoring function that tracks progress towards goals ensuring coherent behavior (imagine being fearful of a stove but not knowing why).
But this isn't true! It has been repeatedly shown that patients without inner brain function react to stimuli (such as being pinched or pricked with a needle) by recoiling from the pain, as do babies with no experience of pain. So qualia and consciousness seem like they have nothing to do with ensuring coherent behavior. To put this another way, your experiences and interactions with the world could be sufficient to associate the stove with danger, but how does that explain why the experience of touching the stove has qualia, as opposed to simply the pain-reaction of a patient without inner brain function or a baby?
Another counterargument is that our brains carry out lots of "coherent" functions "in the dark". Consider, for example, thermoregulation; most of the time, there is no conscious experience associated with it, but yet it is happening constantly and coherently.
Let's simplify it further: to use a famous example, do you believe that a thermostat is conscious? After all, a theremostat is able to coherently regulate its temperature over time in response to changes in its environment.
>But this isn't true! It has been repeatedly shown that patients without inner brain function react to stimuli (such as being pinched or pricked with a needle) by recoiling from the pain, as do babies with no experience of pain.
Yes, reflexive avoidance behavior doesn't require conscious experience. But as the environment of the organism gets more complex, reflexive avoidance behavior isn't sufficient for competence. For an agent in a complex environment, competent damage avoidance requires engaging with negative valence as a cognitive entity to be planned around and weighed against other interests. This requires unification and consciousness.
>Another counterargument is that our brains carry out lots of "coherent" functions "in the dark". Consider, for example, thermoregulation
This isn't an example of coherent behavior in the sense being used here. The issue is one of voluntary behavior being coherently executed as to achieve some goal without undermining itself.
> But as the environment of the organism gets more complex, reflexive avoidance behavior isn't sufficient for competence. For an agent in a complex environment, competent damage avoidance requires engaging with negative valence as a cognitive entity to be planned around and weighed against other interests. This requires unification and consciousness.
But why does engaging with negative valence, planning, and weighing actions against other interests require subjective experience? That sounds simply like a mathematical function (perhaps using our own past experiences as inputs). Reinforcement Learning is a great counterexample here: AI systems weigh negative valence and execute long-term plans without any qualia.
If thermoregulation is too "reflexive" for you, consider that there are many examples in which humans are able to perform very complex tasks in the absence of qualia. Consider, for instance, the phenomena of highway hypnosis, blindsight or sleepwalking - humans can do incredibly complicated things without qualia.
> This isn't an example of coherent behavior in the sense being used here. The issue is one of voluntary behavior being coherently executed as to achieve some goal without undermining itself.
This argument is circular. The original claim is that behaving coherently in a a complex environment requires consciousness. By shifting the goalposts to say that only voluntary behaviors qualify, you are begging the question. The entire notion of "voluntary" implies conscious intent, so your argument has become "consciously willed behaviors require consciousness".
>But why does engaging with negative valence, planning, and weighing actions against other interests require subjective experience?
I have a few different answers here. None are rock solid. Lets take it as a given that planning requires a unified representation of all inputs to the planning apparatus. Now, going with the example from earlier: an organism touches a hot stove and recoils. We can imagine this behavior without any accompanying qualia. But to plan subsequent behavior around the hot stove, the damaging hotness must be represented in the unified representation in a way that intrinsically carries the semantics of negative valence. Phenomenal pain just is "semantics of negative valence featured in a unified representation". My claim is that this is a conceptual identity; you can't have one without the other. This gives the planning apparatus competence at engaging with signals of bodily damage.
Without intrinsic semantics/phenomenality all you have is a signal with no intrinsic meaning and some context to select behavior downstream of the signal. But planning in dynamic environments requires much more flexible signaling than this kind of static context can provide.
>AI systems weigh negative valence and execute long-term plans without any qualia.
AI systems are highly fragmented representations. It's why you can get them to contradict themselves in the same session, or even one sentence after another. They are not an exemplar of coherent behavior. There's also no negative valence in LLMs. At most they have a representation of good/bad and this spectrum influences the valence/quality/alignment in their behavior. But valence as such is external to the LLM.
>consider that there are many examples in which humans are able to perform very complex tasks in the absence of qualia. Consider, for instance, the phenomena of highway hypnosis, blindsight or sleepwalking - humans can do incredibly complicated things without qualia.
Complexity is relative. The complexity of tasks sans qualia are always starkly deficient compared to comparable tasks with qualia. A wide look at cognitive science demonstrates the inherent value of qualia to highly complex tasks or tasks executed over long timescales.
>This argument is circular. The original claim is that behaving coherently in a a complex environment requires consciousness. By shifting the goalposts...
The goalposts aren't shifted, I'm clarifying the target of the term behavior as there was clearly a disagreement in meaning.
>to say that only voluntary behaviors qualify, you are begging the question. The entire notion of "voluntary" implies conscious intent, so your argument has become "consciously willed behaviors require consciousness".
This misunderstands the debate. The philosophical issue of consciousness is how to explain consciousness given the in principle completeness of physical descriptions and their categorical distinction from phenomenal descriptions. In this context, voluntary behavior is just higher order/complex behavior, it is not taken as downstream of consciousness in principle. There is a parallel conversation in psychology/cognitive science where consciousness is largely understood as wakefulness, attention, reportability, intentionality, etc. In this context "consciousness" (in this restricted sense) is a pre-requisite of voluntary behavior. But that's neither here nor there with regards to the philosophical debate.
I don't think that it is appropriate to use "gestalt" here. The word used in the field is "qualia", it has a precise meaning and is precisely what Nagel was writing about. Gestalt, to my understanding, is quite different, even when used in english psychology writing.
My usage of gestalt isn't without precedent[1]. I like gestalt better than qualia as a neutral description of the explanandum. Qualia is an atomistic view of consciousness and so is heavily theory-laden. I had just read a comment from the previous thread[2] on how this paper was translated into other languages and the lack of an equivalent "what its like" phrasing. The translations struck me as missing the virtue of the what it's like phrasing, namely identifying the intrinsic perspectivalness of cognitive systems without taking a stand on how to cash it out. I was trying to think of a better phrasing that could translate well and I landed on gestalt.
> This gestalt isn't just for biological organisms, but any system for which its decision making engages with representations of the external environment unified with a self-representation to form a coherent representation of a persistent entity engaged with an external world.
This doesn't seem quite right, or at least underspecified. We can talk about this stuff concretely these days, at least in the context of digital systems. E.g. i can draw up a diagram of a system that takes in some camera and audio data (and tactile, proprioceptive, etc.), tokenizes it then runs that + past state data through some autoregressive VLM to drive an inference process. The state being passed around can be written out analytically for a given trained model - the external and internal environmental representations, the linear algebra that transforms them into latent action representations, the process by which that is transformed into control signals. It seems difficult to claim that the computational process that implements this has any more or less of a gestalt then one multiplying two matrices together. So it's not just the existence of certain representations or computational loops that seems to lead to possessing a gestalt.
> It seems difficult to claim that the computational process that implements this has any more or less of a gestalt then one multiplying two matrices together. So it's not just the existence of certain representations or computational loops that seems to lead to possessing a gestalt.
I've thought a lot about what is lacking in modern VLMs that preclude consciousness. In my view the difference is that their talk of "self" is a simulacrum of the real thing. Current models are feed forward and so self-talk is driven by some parameter that turns on when the network detects context that possibly references the model, and this parameter drives downstream self-talk. It's a very good simulacrum, but it is a far cry from a model with recurrent self-reference around which the inference process is organized. The richness of the self-model in a hypothetical recurrent network with capabilities of modern LMs is much greater than the parameter on/off representation in feed forward networks.
Completely agree. This is what Hofstadter means by a strange loop. Our current LLMs have no attentional autonomy by design. The recursion is superficial and without its own Now. Adding attentional autonomy is The frightening alignment issue.
Seems like a rather ad hoc restriction. The issue is one of inferring the structure of the processes generating the output. I suppose given enough time and an adversarial style of interaction one could in principle determine the computational structure of any system with high confidence. So probably yes, modulo real-world concerns.
Why do such systems need this gestalt? Why consciousness instead of everything happening in the dark? The recognition of oneself as situated in the world is crucial to coherent engagement with the world. It is how an entity can ensure its body parts are moving towards the same goal. It's how behavior over time doesn't undermine its purpose. Fragmented, incoherent behavior does not serve self-preservation.
LLMs as they are currently constructed probably aren't conscious, but we are a hop skip and a jump away from ones that are.