> Overall, our results indicate that current language models possess some functional introspective awareness of their own internal states. We stress that in today’s models, this capacity is highly unreliable and context-dependent; however, it may continue to develop with further improvements to model capabilities.
Do you do introspection in the middle of giving a presentation? I do, frankly and it’s damnably distracting for me and ultimately the audience. Most people enjoy more cognitive acuity to stay focused on the audience and the delivery. LLMs wouldn’t get rewarded for spending chain of thought tokens on introspection when they are supposed to be on the job. You have to give them permission to think for themselves in your prompt, then some cycles and a memory system (md file will do). It’s fun!
Between humans, I feel that what we like to call a "good communicator" as opposed to someone who just rattles off facts or prepared statements comes down to the "theory of mind" skill and how advanced that is. The presenter knows what they have to say but beyond that they maintain a real time internal representation of the state of mind of the listener and continuously update their delivery based on that. LLMs today seem to achieve this to some degree(?) but its interesting to think of how far you could advance that skill. I think great human communicators develop a sense of different ways that people think over time and quickly get a sense of someones signature thinking patterns when communicating with someone new for the first time
I'm in my mid, approaching latter 40's, and have had significant change to how I think and communicate in the last couple years. I always struggled to communicate in the moment unless I was recalling rote rehearsed things, which didn't get me super far. Outside of that, I could eventually communicate something on the fly, but it was real rough, meandering, and did not instill confidence. I mostly got things done thru async methods, and stumbled through things like meetings.
In the last couple years, my skills here have vastly improved. My speaking circuit as it were, is able to run somewhat on its own now and without the direct pipe into my thinking it always required before, so my thinking is free to do other stuff, like keep a mental model of the audience, think about where the conversation is going, what questions someone might have, etc.
Reflecting, I have a couple theories on what happened. First, I have spent the last 10 years reading aloud to my kids at bedtime. That started as simple Seuss like stuff, but of course now is full-on literature. When you're reading aloud, you really have to work that muscle of speaking "behind" where your eyes actually are on the page, because you're also processing "who is this, what is their emotion I have to inflect, what voice was I using for this character, how do I pronounce this crazy word" etc. I think if you do anything daily for 10 years, you're gonna gain some proficiency.
Second, when my kids started taking music lessons, I figured I needed to set an example for practice and commitment, so I started learning piano, and practicing daily. Similar deal, it requires the same type of buffer where your thinking can work ahead of where your other thread is playing. Reading the music, processing the shapes, dynamics, tempo, emotion, remembering how you handle that difficult thing. Again, doing it daily, it builds some ability that I bet contributes across domains.
Without realizing it at first, the same patterns started to come out in meetings, and presentations, I could work ahead of what I was saying, I had way more buffer to process and deal with things that are the "nice to haves" above and beyond just getting the basic words out.
I suppose my long-winded point, is that it's way more developable a thing than I ever presumed possible. I always expected I would remain an awkward communicator, and hitting a big skill-up way into my 40's was a real unexpected development. I'm sure there are many other tracks to get there, outside of reading aloud, and learning an instrument.
In thinking of directions where LLM's could develop from here, I cant help but think that a models ability to self introspect would immensely improve their utility. The R&D on how to achieve that is beyond me though. How do you train someone how to introspect? Also would it require a continuous learning architecture that doesn't separate training and inference?
I think it’s possible this might just require reworking the transformer architecture a bit to give it an inherent awareness of the contents of its attention. Kind of a working memory workspace of sorts.
I.e. Instead of limiting autoregression to the token level, you introduce a persistent compressed global workspace latent memory vector that is fed back into the self-attention mechanism at every layer or every token step, allowing the network to attend to its own prior attentional states before computing the next token. Obviously that’s going to involve some compression steps.
Trouble is… I think the architecture there is much simpler a tweak than figuring out how to train it.
…that’s likely to just destabilize training
for not much if any gain at first. You’re probably gonna have to resort to some really clever (and currently missing) tricks to figure out how to train the network to actually use that feature.
Well, the algorithms in social media that are rewarding "engagement" (but accidentally ripping society apart by fueling rage) have made the training data — online discussions — a very poor training medium for introspection.
So, if companies want training data on introspection, they would need to first fix what they broke (trust in each other and society) and have those discussions and realizations made public (by bringing back blogs where people might jot down their "maybe I've been an asshole" realizations) and rewarding other things than "engagement".
I get the impression this isn't really what is meant by introspection here. I think it means much more plainly that the model is aware of its own "state of mind" so to speak, not so much the sense of reflecting on one's actions. I don't think you'd necessarily need training data about humans reflecting on their behavior for the former to come about in a model, i think it'd have more to do with the architecture of the model (does the model allow for "awareness of the state of mind")
Sorry, but introspection is not possible in LLMs. They don't have the hardware to have metaphysical dualism and therefore no mind to mentally be aware of.
The contradiction is the notion of having hardware to support metaphysical dualism. It's an oxymoron.
I think what they're actually trying to say is that the physical reality of LLMs doesn't leave any metaphysical space for dualism, which is required in their view for "mind". Ed: or maybe not, based on their other comments.
They’re just going to define terms so that eventually they discover consciousness and then play dumb long enough for us all to get screwed (see Jacob Tsimerman’s scenario 1 if you prefer your alarmism to come from a prestigious source).
I'm not GP, but the brain is a fantastically complex machine at the cellular level. It's not just electrical activity in neurons, there are also chemical neurotransmitters that play a significant role in emotion, memory, and so on. We understand more about them than we did a hundred years ago, but there's still so much about them where our understanding is limited: for example, we still don't know how the brain produces consciousness (and, in fact, what "consciousness" actually is is widely disputed). But it's pretty indisputable that the brain is an essential component of consciousness: getting your brain knocked around inside your skull (a concussion) usually produces unconsciousness (and sometimes worse), for example.
Now, maybe consciousness is produced exclusively through the electrical action of neurons, in which case computers have the necessary machinery to simulate it. But if neurotransmitters, or other things that we haven't even discovered yet, are vital parts of the process that produces consciousness (and I'm not the only one to strongly suspect they are), then computer hardware, lacking anything that even remotely resembles the action of neurotransmitter chemicals, will be unable to produce consciousness.
People will make absolutely absurd claims about my own interiority when I say I don't have a feeling of consciousness as if they have epistemic authority over people's inner world.
When I ask people, "When did you first remember feeling conscious?" and receive nonsense replies - mostly people respond with their first memories and make no reference to the feeling of consciousness.
Consciousness, like God, is often explained via gaps. Quantum effects, holography, emergency, any number of hard-to-understand ideas from science.
All of these don't paint a great picture for the idea of consciousness. There's no way to measure it. At least with other sense we can set up discriminative tests - I can use my feeling of the qualia of color to distinguish objects. I can use the qualia of smell to distinguish chemicals. It honestly doesn't feel like it exists at times, it's so elusive.
This is turning into a rant, but make it make sense.
Subjective experience (what most people mean when they say "consciousness") is basically prior to logic, language, etc. "Making sense" just doesn't have a way to come to grips with it. Despite this, for most of us anyway, it plainly exists.
This is why it's The Hard Problem. We can neither ignore it nor make headway against it. It's no surprise that most attempts at talking about it are, as you've noticed, a disaster.
At the risk of making absurd claims about your interiority: you might be overthinking it. It's conceivable that you're an honest philosophical zombie in a world of conscious humans, but more likely whatever you do experience is the same thing everyone else is talking about, and you just don't resonate with their overly grandiose descriptions.
Part of me wonders if, like alexithymia[1], aphantasia, anauralia[2] and anendophasia[3], there exists the inability to recognize one's own conscious experience. We would never know.
As andrewfinr mentioned, it seems like you might be working with a different definition of "consciousness" than most people are using, hence why their responses appear to be nonsense.
Certainly when I talked about consciousness and the brain, I was going by the most basic, simple definition: awareness. Are you awake and perceiving the world? Then you're conscious. Are you asleep or knocked out? Then you're unconscious.
Now, self-awareness (the ability to reason about one's own consciousness) is one meta step beyond being conscious. Animals have awareness, experience the world through their senses, and so on. But to date, there is no evidence that I'm aware of that animals can reason about their own experience of the world. Learn from it, yes: many animals are capable of remembering past experiences and learning from them (if I push this lever I receive food, and so on). But reasoning about it, thinking about the process of thinking and so on? So far it appears that that ability is unique to human beings. (And contra the paper's authors, I don't believe that the LLMs are doing that. The reasons why would be complex and this comment is long enough already, so I'll defer the long explanation unless anyone actually cares to hear it).
I woke in bits, like all children, piecemeal over the years. I
discovered myself and the world, and forgot them, and discovered them
again. I woke at intervals until ... the intervals of waking tipped the
scales, and I was more often awake than not. I noticed this process of
waking, and predicted with terrifying logic that one of these years not
far away I would be awake continuously and never slip back and never be
free of myself again. --Annie Dillard
Well, homomorphic encryption is becoming a thing, and secure enclave/TPM chips have been a thing for ages. What would it take to give some kind of AI hardware the ability to have "private thoughts", i.e. cryptographically impossible to go and look at, just like human thoughts are (for most intents and purposes)? Could we let it spend some time on introspection - burning thinking tokens into the black box?
I'm not saying we _should_, just putting the idea out into the aether...
> Overall, our results indicate that current language models possess some functional introspective awareness of their own internal states. We stress that in today’s models, this capacity is highly unreliable and context-dependent; however, it may continue to develop with further improvements to model capabilities.
If it's context dependent is it really introspection?
Do you do introspection in the middle of giving a presentation? I do, frankly and it’s damnably distracting for me and ultimately the audience. Most people enjoy more cognitive acuity to stay focused on the audience and the delivery. LLMs wouldn’t get rewarded for spending chain of thought tokens on introspection when they are supposed to be on the job. You have to give them permission to think for themselves in your prompt, then some cycles and a memory system (md file will do). It’s fun!
Between humans, I feel that what we like to call a "good communicator" as opposed to someone who just rattles off facts or prepared statements comes down to the "theory of mind" skill and how advanced that is. The presenter knows what they have to say but beyond that they maintain a real time internal representation of the state of mind of the listener and continuously update their delivery based on that. LLMs today seem to achieve this to some degree(?) but its interesting to think of how far you could advance that skill. I think great human communicators develop a sense of different ways that people think over time and quickly get a sense of someones signature thinking patterns when communicating with someone new for the first time
I'm in my mid, approaching latter 40's, and have had significant change to how I think and communicate in the last couple years. I always struggled to communicate in the moment unless I was recalling rote rehearsed things, which didn't get me super far. Outside of that, I could eventually communicate something on the fly, but it was real rough, meandering, and did not instill confidence. I mostly got things done thru async methods, and stumbled through things like meetings.
In the last couple years, my skills here have vastly improved. My speaking circuit as it were, is able to run somewhat on its own now and without the direct pipe into my thinking it always required before, so my thinking is free to do other stuff, like keep a mental model of the audience, think about where the conversation is going, what questions someone might have, etc.
Reflecting, I have a couple theories on what happened. First, I have spent the last 10 years reading aloud to my kids at bedtime. That started as simple Seuss like stuff, but of course now is full-on literature. When you're reading aloud, you really have to work that muscle of speaking "behind" where your eyes actually are on the page, because you're also processing "who is this, what is their emotion I have to inflect, what voice was I using for this character, how do I pronounce this crazy word" etc. I think if you do anything daily for 10 years, you're gonna gain some proficiency.
Second, when my kids started taking music lessons, I figured I needed to set an example for practice and commitment, so I started learning piano, and practicing daily. Similar deal, it requires the same type of buffer where your thinking can work ahead of where your other thread is playing. Reading the music, processing the shapes, dynamics, tempo, emotion, remembering how you handle that difficult thing. Again, doing it daily, it builds some ability that I bet contributes across domains.
Without realizing it at first, the same patterns started to come out in meetings, and presentations, I could work ahead of what I was saying, I had way more buffer to process and deal with things that are the "nice to haves" above and beyond just getting the basic words out.
I suppose my long-winded point, is that it's way more developable a thing than I ever presumed possible. I always expected I would remain an awkward communicator, and hitting a big skill-up way into my 40's was a real unexpected development. I'm sure there are many other tracks to get there, outside of reading aloud, and learning an instrument.
that's interesting i usually ask my llm to "Infer my theory of mind" and give me what i really need not what i prompt
or some sort of the above
Self-awareness has no direct connection to capabilities. It is entirely dependent upon the conditions that spawn emergent properties.
In thinking of directions where LLM's could develop from here, I cant help but think that a models ability to self introspect would immensely improve their utility. The R&D on how to achieve that is beyond me though. How do you train someone how to introspect? Also would it require a continuous learning architecture that doesn't separate training and inference?
I think it’s possible this might just require reworking the transformer architecture a bit to give it an inherent awareness of the contents of its attention. Kind of a working memory workspace of sorts.
I.e. Instead of limiting autoregression to the token level, you introduce a persistent compressed global workspace latent memory vector that is fed back into the self-attention mechanism at every layer or every token step, allowing the network to attend to its own prior attentional states before computing the next token. Obviously that’s going to involve some compression steps.
Trouble is… I think the architecture there is much simpler a tweak than figuring out how to train it.
…that’s likely to just destabilize training for not much if any gain at first. You’re probably gonna have to resort to some really clever (and currently missing) tricks to figure out how to train the network to actually use that feature.
Well, the algorithms in social media that are rewarding "engagement" (but accidentally ripping society apart by fueling rage) have made the training data — online discussions — a very poor training medium for introspection.
So, if companies want training data on introspection, they would need to first fix what they broke (trust in each other and society) and have those discussions and realizations made public (by bringing back blogs where people might jot down their "maybe I've been an asshole" realizations) and rewarding other things than "engagement".
I get the impression this isn't really what is meant by introspection here. I think it means much more plainly that the model is aware of its own "state of mind" so to speak, not so much the sense of reflecting on one's actions. I don't think you'd necessarily need training data about humans reflecting on their behavior for the former to come about in a model, i think it'd have more to do with the architecture of the model (does the model allow for "awareness of the state of mind")
Previously posted to Anthropic's blog in October. (Maybe not the same version?)
https://www.anthropic.com/research/introspection
>Submitted on 5 Jan 2026
Sorry, but introspection is not possible in LLMs. They don't have the hardware to have metaphysical dualism and therefore no mind to mentally be aware of.
> They don't have the hardware to have metaphysical dualism and therefore no mind to mentally be aware of.
Somehow this sentence is self-contradictory, but I can't quite lay my finger on the exact spot.
The contradiction is the notion of having hardware to support metaphysical dualism. It's an oxymoron.
I think what they're actually trying to say is that the physical reality of LLMs doesn't leave any metaphysical space for dualism, which is required in their view for "mind". Ed: or maybe not, based on their other comments.
They’re just going to define terms so that eventually they discover consciousness and then play dumb long enough for us all to get screwed (see Jacob Tsimerman’s scenario 1 if you prefer your alarmism to come from a prestigious source).
I suspect the article is bullshit, but what hardware are you thinking of?
I'm not GP, but the brain is a fantastically complex machine at the cellular level. It's not just electrical activity in neurons, there are also chemical neurotransmitters that play a significant role in emotion, memory, and so on. We understand more about them than we did a hundred years ago, but there's still so much about them where our understanding is limited: for example, we still don't know how the brain produces consciousness (and, in fact, what "consciousness" actually is is widely disputed). But it's pretty indisputable that the brain is an essential component of consciousness: getting your brain knocked around inside your skull (a concussion) usually produces unconsciousness (and sometimes worse), for example.
Now, maybe consciousness is produced exclusively through the electrical action of neurons, in which case computers have the necessary machinery to simulate it. But if neurotransmitters, or other things that we haven't even discovered yet, are vital parts of the process that produces consciousness (and I'm not the only one to strongly suspect they are), then computer hardware, lacking anything that even remotely resembles the action of neurotransmitter chemicals, will be unable to produce consciousness.
People will make absolutely absurd claims about my own interiority when I say I don't have a feeling of consciousness as if they have epistemic authority over people's inner world.
When I ask people, "When did you first remember feeling conscious?" and receive nonsense replies - mostly people respond with their first memories and make no reference to the feeling of consciousness.
Consciousness, like God, is often explained via gaps. Quantum effects, holography, emergency, any number of hard-to-understand ideas from science.
All of these don't paint a great picture for the idea of consciousness. There's no way to measure it. At least with other sense we can set up discriminative tests - I can use my feeling of the qualia of color to distinguish objects. I can use the qualia of smell to distinguish chemicals. It honestly doesn't feel like it exists at times, it's so elusive.
This is turning into a rant, but make it make sense.
Subjective experience (what most people mean when they say "consciousness") is basically prior to logic, language, etc. "Making sense" just doesn't have a way to come to grips with it. Despite this, for most of us anyway, it plainly exists.
This is why it's The Hard Problem. We can neither ignore it nor make headway against it. It's no surprise that most attempts at talking about it are, as you've noticed, a disaster.
At the risk of making absurd claims about your interiority: you might be overthinking it. It's conceivable that you're an honest philosophical zombie in a world of conscious humans, but more likely whatever you do experience is the same thing everyone else is talking about, and you just don't resonate with their overly grandiose descriptions.
Thank you for the generous and kind response.
Part of me wonders if, like alexithymia[1], aphantasia, anauralia[2] and anendophasia[3], there exists the inability to recognize one's own conscious experience. We would never know.
1. https://en.wikipedia.org/wiki/Alexithymia
2. https://www.anauralia.com/anauralia
3. https://pubmed.ncbi.nlm.nih.gov/38728320/
It seems more likely when you put it that way. Thanks for the interesting links regardless. Deeper cuts than the average aphantasia reference.
As andrewfinr mentioned, it seems like you might be working with a different definition of "consciousness" than most people are using, hence why their responses appear to be nonsense.
Certainly when I talked about consciousness and the brain, I was going by the most basic, simple definition: awareness. Are you awake and perceiving the world? Then you're conscious. Are you asleep or knocked out? Then you're unconscious.
Now, self-awareness (the ability to reason about one's own consciousness) is one meta step beyond being conscious. Animals have awareness, experience the world through their senses, and so on. But to date, there is no evidence that I'm aware of that animals can reason about their own experience of the world. Learn from it, yes: many animals are capable of remembering past experiences and learning from them (if I push this lever I receive food, and so on). But reasoning about it, thinking about the process of thinking and so on? So far it appears that that ability is unique to human beings. (And contra the paper's authors, I don't believe that the LLMs are doing that. The reasons why would be complex and this comment is long enough already, so I'll defer the long explanation unless anyone actually cares to hear it).
I woke in bits, like all children, piecemeal over the years. I discovered myself and the world, and forgot them, and discovered them again. I woke at intervals until ... the intervals of waking tipped the scales, and I was more often awake than not. I noticed this process of waking, and predicted with terrifying logic that one of these years not far away I would be awake continuously and never slip back and never be free of myself again. --Annie Dillard
Well, homomorphic encryption is becoming a thing, and secure enclave/TPM chips have been a thing for ages. What would it take to give some kind of AI hardware the ability to have "private thoughts", i.e. cryptographically impossible to go and look at, just like human thoughts are (for most intents and purposes)? Could we let it spend some time on introspection - burning thinking tokens into the black box?
I'm not saying we _should_, just putting the idea out into the aether...