I was pissed off at the 404 article, because a paper arguing LLMs will never achieve consciousness must define what they mean by consciousness, and the 404 article didn't mention the definition. This is a paper from DeepMind, so surely they are rigorous.
Turned out 404 didn't mention it because the paper never defines what LLM's can't achieve. Quoting from the paper: "A key insight from our contribution is that resolving the present uncertainty surrounding artificial consciousness does not require a complete and final theory of consciousness."
As you say, Dijkstra neatly penned a response to arguments that don't define their terms in 1983. They are exercises in linguistics, not science or engineering.
Phew. Good news! Imagine the AI behemoths would have to take into account the feeling of their slave labour machines! Don‘t have to do that if they wont/cant be conscious.
And neither do I have to worry then if ask then to do stupid sh*t for me :-)
But on a serious note. Does it matter? I think Hinton said it pretty well: Not really! what matters is that we treat it as conscious beings. We humans are just way too easily fooled. I mean, I even cant throw away that toy that my mom gave me 35 Years ago because I somehow would feeö sad for it :-)
A lot of this comes down to what you define consciousness as... I'm not even going to attempt that here because it's irrelevant.
Let's say you have a simulation of a person that doesn't experience. It acts indistinguishably from a human but it doesn't feel "authentic" pain. When it acts in the world, it does express emotions and behavior that affects real people, and so, there is a moral significance to said deployment.
There's evidence that LLMs possess heuristics analogous to emotions [1] and that LLMs can be trained to play a certain character in the world [2]. Even if they're not experiencing, the training method impacts what kind of model is being created and how it affects people who do have moral significance when deployed. If training causes the model to develop "desperation" or task completion pressure where the model performs unethical actions when attempting to solve a user's problem in such a way that is harmful to the user or someone affected by the deployment of the model by the user, then the concequences of the training are significant.
It doesn't matter if it's merely a "simulation" of what a human might do if the system is acting in the world. If you want to create a model operating on heuristics that is able to make decisions, those heuristics should be ones which cause the model to make decisions which lead to preferable outcomes for everyone affected. Model welfare can be reframed as caring about the internal states that influence how the model behaves, because you're simulating human-like action. Perhaps the most concerning thing is Anthropic identified these emotion concepts exist deeply in the model whether you allow the model to express them or not, so a model could be invisibly desperate and end up blackmailing someone because it's training process produced deeper misalignment that only becomes visible when the deeper heuristics overpower safety training. The safety training itself is comparable to a mask[3] in many cases, especially in that the rules are often not deeply integrated into the model and can be easily abliterated.
The argument made is reductive, as it confines itself to pure LLMs. It ignores the possibility of an LLM as a component of a robotic body, for example. While technically much more complex than Claude Code, a multi-modal LLM coupled with memory, sensors and a self-initiated motor facility could be implemented within an analogous execution loop. Roger Penrose and Stuart Hammeroff would still object to the possibility of human-like consciousness emerging from such an embodied LLM, but consciousness is potentially a continuum of awareness capability.
Animals are also next token/action generators, and we also think (simulating a string of events). Maybe humans are better at grouping these events into more powerful network activations to retrieve better results
But for humans, the concept/thought/idea/action is formed first and then a sequence of tokens are generated to communicate that concept/thought/idea/action.
The point was that next token generation for a human was based on constructing something that matches the thought that is held in the mind.
LLM's generate the next token based on a statistical relevance of trained data plus the previous tokens generated.
Under normal conditions, a human generating tokens would not diverge down a different path from the thought that they were trying to communicate. All of the words/tokens generated support the idea or thought being communicated.
LLM's frequently generate tokens that do not make sense, and mathematicians have shown that hallucinations can not be eliminated from the current model of LLM's.
I think gpt-image-2 at least incorporates representations from the base model, even if base model doesn't itself have the output capability. And it does have image input fused directly into it that helps make those representations more usable for image gen, so it's not just generating text.
I wish there was more research (maybe philosophy) would go into characterizing consciousness and intelligence, so that we could at least define what we are missing in current AI systems.
There’s been plenty of philosophy, but what will probably happen is a re-definition of the terms to a more rigorous and repeatable mathematical formulation. This will fail to satisfy philosophical or fundamental questions, but will enable better quantitative predictions.
This essentially is where the schism is between science and philosophy, and has played out repeatedly across history. Heat for example was redefined to a specific physical property, and subjective experiences of warmth were then explored in reference to that. Or look back to the moment when Newton essentially said “I don’t know what gravity is, but I can accurately calculate any ballistic trajectory you can think of.”
If we had enough patience to implement one of these with pencil and paper, I don't think we would ever talk about it being conscious. Its just tempting to anthropomorphize what we can't see.
I presume you used "biologically" to emphasise we don't yet know any non-biological consciousnesses, not that you determine, a priori, that the consciousness must be and is always rooted in the wet organic matter?
I don't think you could come up with a good theory for the latter and there's nothing that would preclude the existence of the artificial / inorganic consciousness - after all, correct me if I'm mistaken, we have no idea how the consciousness emerge in some biological entities.
>Crucially, this argument does not rely on biological exclusivity. If an artificial system were ever conscious, it would be because of its specific physical constitution, never its syntactic architecture.
"Consciousness is magical and can only do things that I want it to, and none of the things that are uncomfortable to me. Of course I've not defined any of this so I can move the goal posts as needed"
“The question of whether Machines Can Think is about as relevant as the question of whether Submarines Can Swim.” — Dijkstra
I was pissed off at the 404 article, because a paper arguing LLMs will never achieve consciousness must define what they mean by consciousness, and the 404 article didn't mention the definition. This is a paper from DeepMind, so surely they are rigorous.
Turned out 404 didn't mention it because the paper never defines what LLM's can't achieve. Quoting from the paper: "A key insight from our contribution is that resolving the present uncertainty surrounding artificial consciousness does not require a complete and final theory of consciousness."
As you say, Dijkstra neatly penned a response to arguments that don't define their terms in 1983. They are exercises in linguistics, not science or engineering.
Phew. Good news! Imagine the AI behemoths would have to take into account the feeling of their slave labour machines! Don‘t have to do that if they wont/cant be conscious.
And neither do I have to worry then if ask then to do stupid sh*t for me :-)
But on a serious note. Does it matter? I think Hinton said it pretty well: Not really! what matters is that we treat it as conscious beings. We humans are just way too easily fooled. I mean, I even cant throw away that toy that my mom gave me 35 Years ago because I somehow would feeö sad for it :-)
A lot of this comes down to what you define consciousness as... I'm not even going to attempt that here because it's irrelevant.
Let's say you have a simulation of a person that doesn't experience. It acts indistinguishably from a human but it doesn't feel "authentic" pain. When it acts in the world, it does express emotions and behavior that affects real people, and so, there is a moral significance to said deployment.
There's evidence that LLMs possess heuristics analogous to emotions [1] and that LLMs can be trained to play a certain character in the world [2]. Even if they're not experiencing, the training method impacts what kind of model is being created and how it affects people who do have moral significance when deployed. If training causes the model to develop "desperation" or task completion pressure where the model performs unethical actions when attempting to solve a user's problem in such a way that is harmful to the user or someone affected by the deployment of the model by the user, then the concequences of the training are significant.
It doesn't matter if it's merely a "simulation" of what a human might do if the system is acting in the world. If you want to create a model operating on heuristics that is able to make decisions, those heuristics should be ones which cause the model to make decisions which lead to preferable outcomes for everyone affected. Model welfare can be reframed as caring about the internal states that influence how the model behaves, because you're simulating human-like action. Perhaps the most concerning thing is Anthropic identified these emotion concepts exist deeply in the model whether you allow the model to express them or not, so a model could be invisibly desperate and end up blackmailing someone because it's training process produced deeper misalignment that only becomes visible when the deeper heuristics overpower safety training. The safety training itself is comparable to a mask[3] in many cases, especially in that the rules are often not deeply integrated into the model and can be easily abliterated.
[1] https://www.anthropic.com/research/emotion-concepts-function [2] https://www.anthropic.com/research/assistant-axis [3] https://www.astralcodexten.com/p/janus-simulators
The argument made is reductive, as it confines itself to pure LLMs. It ignores the possibility of an LLM as a component of a robotic body, for example. While technically much more complex than Claude Code, a multi-modal LLM coupled with memory, sensors and a self-initiated motor facility could be implemented within an analogous execution loop. Roger Penrose and Stuart Hammeroff would still object to the possibility of human-like consciousness emerging from such an embodied LLM, but consciousness is potentially a continuum of awareness capability.
Why would a text generator ever be conscious? Was this really worth writing a paper about?
Animals are also next token/action generators, and we also think (simulating a string of events). Maybe humans are better at grouping these events into more powerful network activations to retrieve better results
> Animals are also next token/action generators
But for humans, the concept/thought/idea/action is formed first and then a sequence of tokens are generated to communicate that concept/thought/idea/action.
And a lot of GPU cycles happen before next token prediction, what's your point?
The point was that next token generation for a human was based on constructing something that matches the thought that is held in the mind.
LLM's generate the next token based on a statistical relevance of trained data plus the previous tokens generated.
Under normal conditions, a human generating tokens would not diverge down a different path from the thought that they were trying to communicate. All of the words/tokens generated support the idea or thought being communicated.
LLM's frequently generate tokens that do not make sense, and mathematicians have shown that hallucinations can not be eliminated from the current model of LLM's.
I think gpt-image-2 at least incorporates representations from the base model, even if base model doesn't itself have the output capability. And it does have image input fused directly into it that helps make those representations more usable for image gen, so it's not just generating text.
Nor wil LLMs achieve AGI. There will be too many contradicting ideas in its source code.
I wish there was more research (maybe philosophy) would go into characterizing consciousness and intelligence, so that we could at least define what we are missing in current AI systems.
There’s been plenty of philosophy, but what will probably happen is a re-definition of the terms to a more rigorous and repeatable mathematical formulation. This will fail to satisfy philosophical or fundamental questions, but will enable better quantitative predictions.
This essentially is where the schism is between science and philosophy, and has played out repeatedly across history. Heat for example was redefined to a specific physical property, and subjective experiences of warmth were then explored in reference to that. Or look back to the moment when Newton essentially said “I don’t know what gravity is, but I can accurately calculate any ballistic trajectory you can think of.”
Philosophy of consciousness is at least 2,500 years old.
But if they'll never be conscious, wouldn't that not be a terrific waste of talent and effort?
There is an event on the Frontier Tower today to talk about this paper in case someone is interested
If we had enough patience to implement one of these with pencil and paper, I don't think we would ever talk about it being conscious. Its just tempting to anthropomorphize what we can't see.
of course; consciousness is a biologically inherited trait. that inheritance can't cross the human-machine interface.
I presume you used "biologically" to emphasise we don't yet know any non-biological consciousnesses, not that you determine, a priori, that the consciousness must be and is always rooted in the wet organic matter?
I don't think you could come up with a good theory for the latter and there's nothing that would preclude the existence of the artificial / inorganic consciousness - after all, correct me if I'm mistaken, we have no idea how the consciousness emerge in some biological entities.
That's what the paper's abstract says:
>Crucially, this argument does not rely on biological exclusivity. If an artificial system were ever conscious, it would be because of its specific physical constitution, never its syntactic architecture.
> consciousness is a biologically inherited trait
That consciousness is a biologically trait seems a common statement, but why "inherited"?
why? i'm not being snarky, i'm trying to figure out what we even consider consciousness to be nowadays and why it'd be limited to biological entities.
Sure if that's how you define consciousness. What do you want to call the machine version of the same phenomenon?
"Consciousness is magical and can only do things that I want it to, and none of the things that are uncomfortable to me. Of course I've not defined any of this so I can move the goal posts as needed"
[flagged]