AI models can flag internal disturbances. Reliable self-knowledge remains unproven

• Detecting an internal disturbance and naming its content appear to involve largely separate mechanisms.
• Some experiments resist simple explanations; other controls challenge whether correct answers establish introspection.
• Improving an injection-detection test did not produce broader self-knowledge: a targeted intervention worsened other self-report tasks.

A year of experiments has narrowed the case for AI introspection—the ability to access and report on a model’s own internal processing. Language models sometimes flag artificially altered neural activity before their written answers reveal the alteration. What that detection means, and whether it can support trustworthy explanations of ordinary decisions, remain unresolved.

The distinction matters for transparency. A chatbot might correctly detect something unusual yet give an unreliable account of it. Follow-up research now provides both evidence for a nontrivial detection mechanism and a concrete warning against treating better performance on one introspection test as better self-knowledge generally.

Anthropic’s injection experiment, illustrated with a company-reported Claude Opus 4.1 example. The model’s descriptions of how the injection felt were not independently verified. Source: Transformer Circuits.

What the original experiment tested

Jack Lindsey, an Anthropic researcher who joined the company after completing a computational-neuroscience doctorate in 2024, devised the original experiment. His work included methods for identifying and manipulating internal activity associated with particular concepts. Anthropic develops Claude, the language-model family he tested, as a November 4 account of the research explains.

The precedent was Anthropic’s 2024 Golden Gate Bridge demonstration. Amplifying a concept-related pattern made Claude bring up the bridge in unrelated answers. But noticing strange prose after writing it would not establish privileged access to internal activity: the model could read its own output.

In his October 29, 2025 study, Lindsey instead added concept-related activation vectors inside the network and asked whether the model detected an injected “thought” and could identify it. Here, “thought” names an experimental manipulation, not a demonstrated subjective experience.

Success required detecting the injection before mentioning its content. With favorable injection strengths and layers, Claude Opus 4 and 4.1—the most capable models in that study—successfully detected and identified the concept in about 20% of injection trials. The systematic tests used vectors derived from 50 words. In separate controls without injections, Lindsey observed no false positives over 100 trials for each production model.

Those are results for a particular intervention and prompting procedure, not a general accuracy rate for self-knowledge. Most injection trials failed. Strong injections could make output incoherent; some models mentioned the injected concept while denying detection. Lindsey also cautioned that descriptions of how an injection supposedly felt could be invented, and that concept vectors could carry meanings beyond their intended labels.

The study went beyond injection detection. Models could sometimes distinguish injected concepts from input text. In another experiment, injecting a concept into earlier activations made a model more likely to accept an artificially supplied response as its own intended answer. Models also changed concept-related activity when instructed to think, or avoid thinking, about a word. These were different tests; a critique of the first does not automatically settle the others.

Detection survives some controls, while identification comes apart

Kyle Mahowald, a computational linguist at the University of Texas at Austin, and Harvey Lederman, then a philosopher there, investigated whether simpler mechanisms could explain Lindsey’s results. Lederman has since moved to NYU and Anthropic, but had no Anthropic connection when they conducted the work, according to the November account.

Their paper, first submitted in March, replicated the injection experiment in large open-source models and separated detecting an anomaly from identifying its content. Models sometimes detected an injection while making unrelated guesses about what it contained, repeatedly supplying common, concrete concepts such as apples.

Detection needed fewer generated tokens than correct identification. The researchers could also degrade identification without degrading detection. They argued that content-agnostic detection was consistent with leading theories of introspection. The November account describes simpler explanations failing to fully account for the detection behavior under their controls.

Naming the concept is a weaker result than it first appears, because activation steering was designed to influence what a model says. Lindsey’s original study itself treated the initial detection as the more interesting mechanism, rather than assuming every subsequent description reflected a separate act of self-observation.

A circuit offers evidence beyond fluent answers

A March follow-up coauthored by Lindsey, revised in June, examined how detection works inside models whose parameters are available for inspection. Most experiments used the Gemma3-27B instruction model. The team explained that its available interpretability tools made detailed circuit analysis feasible; building comparable tools for other models requires substantial computation.

The researchers traced detection to a two-stage circuit. Early features responded to perturbations and suppressed later features that ordinarily promoted a negative answer. Identifying the concept relied largely on distinct, later mechanisms.

Their controls also challenged a simple explanation in which an injected vector merely pushes answers toward “yes.” Detection-relevant information was distributed across multiple directions, and some pairs of opposite steering directions both triggered detection. Several prompt variants maintained detection with no observed false positives, including variants that removed incentives to discuss the injected concept.

The team found a role for post-training, too. Comparing successive checkpoints of the OLMo-3.1-32B model, supervised fine-tuning did not yield accurate discrimination between injection and control trials. Direct preference optimization—a later stage using preferred and rejected responses—did. Additional training experiments supported the importance of that contrastive structure. This is evidence about tested training procedures and models, rather than proof of a universal route to introspection.

The full paper nevertheless notes important limits: most mechanistic work concerned one model, another language model classified the responses, and generalization beyond artificial concept injection remained unknown.

Macar and colleagues’ schematic of the proposed detection circuit and company-reported intervention results. Gains concern artificial injection tests. Credit: Macar et al., Figure 1, CC BY 4.0. Source: arXiv — Mechanisms of Introspective Awareness, full paper.

Skeptical tests ask what detection actually establishes

A study by NYU language-model researchers Shashwat Singh, Tal Linzen and Shauli Ravfogel, first submitted in May, proposed two requirements for establishing introspection. A test should require privileged access, so input cues alone cannot solve it. It should also distinguish ordinary processing from second-order computation: a representation of the model’s own processing.

Lindsey’s original definition also required an internal metacognitive representation. He explicitly said his study did not demonstrate that representation directly; its experiments were intended to offer indirect evidence.

In the NYU team’s intervention tests, models could not reliably distinguish changes to hidden states from manipulations of input. That supports generic anomaly detection without establishing sensitivity to internal interventions specifically. In a separate paradigm, classifiers given only the input matched models’ predictions of labels derived from their hidden states. Those predictions therefore did not demonstrate privileged internal access.

An August 24 analysis by Zak Miller and Heather Demarest, affiliated respectively with AI Digest and the University of Colorado Boulder, challenged detection from another direction. It tested four publicly accessible models, including GLM-5.2, using 50 concepts, ten trial-number variants and four nonzero injection strengths.

The researchers required one-word answers to two questions: whether the model detected an injection, and whether it was in its normal, unmodified state. Correct detection should produce opposite answers—yes to the first, no to the second. Instead, injections often pulled both responses toward the same yes-or-no preference. Similar shifts occurred on unrelated factual questions.

Their proposed explanation combines each question’s baseline answer preference with a concept-specific value that draws the answer toward it. Tested on held-out questions, the account explained most of the variance in the yes-versus-no preference at every injection strength. That challenges the inference from a correct detection answer to introspection, rather than disputing that injections change computation.

The comparison has limits. Miller and Demarest tested different models and a more constrained response format; they did not test Claude and explicitly called for Anthropic to do so. Lindsey’s original Claude study had tested unrelated yes-or-no questions without finding a general increase in affirmative answers. The competing results cannot be collapsed into a single replication verdict.

Better injection detection did not mean broader self-knowledge

The mechanistic follow-up’s intervention tests also examined ways to elicit more detection. Removing a refusal-associated direction from Gemma3-27B raised detection from 10.8% to 63.8% of injection trials at a lower injection strength. False positives on controls rose from 0% to 7.3%. The authors interpreted the gain as evidence that refusal behavior suppressed genuine detection, while acknowledging that refusal also reduced false alarms.

A separate learned intervention gave a different result. The researchers trained an always-present additive vector using 400 concepts and evaluated it on 100 held-out concepts. It improved injection detection with no observed false positives in that evaluation. That supports their argument that default behavior understates what the model can do on this task.

But the same paper’s downstream tests show why the qualification matters. The intervention made the model less likely to disclose a supplied hint in its reasoning when the hint changed its answer:

These rates measure acknowledgment of answer-changing hints, rather than accuracy on the questions or detection of injected concepts. In a separate evaluation containing 1,900 examples across 19 categories, detection of externally supplied assistant responses also fell, from 36.3% to 16.1%. The model’s responses became shorter.

The authors concluded that the learned vector elicited a narrow behavior and did not transfer to broader self-knowledge. Better injection detection can therefore coexist with worse reporting about other influences on a model’s answers.

Company-reported results show lower acknowledgment of answer-changing hints with the learned vector applied at layer 29. These rates measure disclosure, not question-answering accuracy. Source: arXiv — Mechanisms of Introspective Awareness, full paper.

The next test is whether the skill transfers

Resolving the mechanism dispute requires controls that make rival accounts predict different results under comparable conditions: reversing the response required by the question, separating input anomalies from internal interventions, and testing the same models rather than relying on cross-model comparisons.

For practical transparency, the next hurdle is transfer. Can a mechanism that detects an artificial disturbance also help a model accurately explain influences on its ordinary decisions? The learned-vector results make that an empirical question, not a benefit that follows from a higher detection score.

Nor do these experiments establish subjective experience. Lindsey’s study distinguished functional access to internal activity from consciousness and cautioned against strong inferences. The immediate research task is narrower and testable: establish which self-reports track the underlying computation, under which conditions, and whether improving them survives checks beyond the task used to elicit them.