📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=ig5RNJJaFJE

Inside the Brain of an AI: Mapping the Circuits of Claude 3.5

Recent breakthroughs in “mechanistic interpretability” from Anthropic are pulling back the curtain on how large language models actually process information. By using advanced techniques like circuit training and transcoders, researchers can now pinpoint the exact neural pathways responsible for multi-step reasoning, poetic planning, and even medical diagnosis.

Core Question: How can we interpret and intervene in the internal neural circuits of a transformer to understand the mechanistic logic behind its outputs?

Highlights

• Introduction of Cross-Layer Transcoders (CLTs) to map features across model layers.
• Evidence of internal “planning” when the model writes rhyming poetry before tokens are generated.
• Demonstration of feature “clamping” to manipulate model outputs by swapping internal concepts.
• Discovery of “multilingual neurons” that represent concepts independently of specific languages.

⏱️ Reading time: approx. 7 minutes · Saves you about 49 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

From Sparse Features to Circuit Training

The Evolution of Model Interpretation

Anthropic’s journey into mechanistic interpretability began with identifying single features, like a “Golden Gate Bridge” neuron, using Sparse Autoencoders (SAEs) to map activations in toy models.

While SAEs provided a holistic view of what concepts a model understands, they lacked the resolution to track how those concepts interact across different layers of the neural network. To solve this, researchers moved toward “circuit training,” which involves training a transcoder model to mimic the original model layer-by-layer. This allows us to observe the flow of information through specific “circuits” rather than just looking at isolated snapshots of neuron activity.

This process creates what is known as a Local Replacement Model. It works by freezing the original attention layers and replacing the feed-forward MLP networks with transcoders. By enforcing a sparsity penalty during training, the system ensures that only a few key features fire at any given time, making the complex internal logic much easier for human eyes to parse and understand.

💡 Digging Deeper

Q: Why do researchers freeze the attention layers during this process?
A: Freezing attention ensures the training objective focuses solely on the sparse transcoders without altering the model’s fundamental behavior or creating “randomized” weights.

Q: Is this process computationally expensive?
A: Extremely. Training a transcoder for a 9B parameter model can take up to 4,000 H100 GPU hours, making it a high-resource research endeavor.

Multi-Step Reasoning and Internal Planning

Tracking Logical “Hops” in the Latent Space

Does an LLM simply guess the next token, or is it performing a genuine multi-hop calculation when asked a complex question?

When asked for the capital of the state containing Dallas, the model must identify “Texas” and then retrieve “Austin.” Attribution graphs reveal that features for “Texas” activate in early layers, which then trigger “say a capital” features in later layers. This suggests the model isn’t just pattern-matching a single phrase but is actually passing information through a logical chain of internal concepts to arrive at the final answer.

We see a similar phenomenon in creative writing, particularly when the model is tasked with composing rhyming couplets.

Researchers found that “planning features” for the final rhyming word actually activate several tokens before that word is even written. If the model is writing a poem about a “carrot,” the feature for “rabbit” starts firing at the beginning of the second line, proving the model “looks ahead” internally to ensure the rhyme remains coherent.

💡 Digging Deeper

Q: Can we change the model’s “mind” mid-process?
A: Yes. By “clamping” a specific feature (like forcing the “California” feature to fire instead of “Texas”), the model will change its answer from Austin to Sacramento even if the prompt still mentions Dallas.

Q: Does this prove models “think” like humans?
A: It proves they have a mechanistic representation of logical steps, though the “thought” is still a series of high-dimensional vector transformations rather than conscious deliberation.

Multilingualism and the Hallucination Circuit

Language-Independent Concepts

One of the most fascinating discoveries involves multilingual representations, where the model uses language-independent “supernodes” to represent universal concepts like “large” or “antonym.”

These features fire regardless of whether the input is in English, French, or Chinese. This suggests that the model develops an internal “world model” that transcends the specific vocabulary of any single human language. However, researchers noted an “English bias” where non-English outputs are often mediated through English-leaning features in the middle layers before being translated back to the target language at the output.

This mapping also sheds light on hallucinations and the impact of fine-tuning.

By identifying “Known Answer” features versus “Unknown Name” features, researchers can see how RLHF training encourages the model to trigger a “refusal circuit” when it lacks specific data. When the “Known Answer” circuit misfires on obscure information, the model hallucinate with high confidence. By artificially boosting the “I don’t know” feature, researchers can force the model to refuse a question it would otherwise have lied about.

💡 Digging Deeper

Q: How does the model recognize it doesn’t know something?
A: It identifies “Unknown Name” features that represent unfamiliar strings. This triggers a specific circuit developed during fine-tuning that prioritizes the “Assistant” persona’s refusal protocol.

Q: Can we use this to stop hallucinations entirely?
A: Theoretically, yes. If we could detect the “Known Answer” feature firing for a false fact, we could suppress it, though the real-time computational overhead is currently too high for production.

Key Takeaways

Mechanistic interpretability is moving beyond identifying isolated neurons to mapping entire functional circuits. This transition allows us to see how models perform logic, plan creative tasks, and navigate multilingual concepts. It effectively turns the “black box” of AI into a navigable map of logical pathways, providing a structural basis for understanding model behavior.

The ability to intervene via “clamping” is a game-changer for AI safety and control. By boosting or suppressing specific features, we can force models to refuse harmful queries or correct hallucinated facts at a structural level. However, the high computational cost of training these replacement models—often thousands of GPU hours—remains a significant barrier to widespread adoption in the open-source community.

Q&A

Q1: What is a “transcoder” in this context?
A1: A transcoder is a specific model architecture trained to replace the MLP (feed-forward) layers of a transformer, designed to be sparse and human-interpretable while matching the original model’s output.

Q2: Does Claude 3.5 Haiku use English as an internal “bridge” for other languages?
A2: Yes, evidence suggests that while it has multilingual neurons, English is often “privileged,” meaning non-English concepts are sometimes processed through English-leaning representations in middle layers.

Q3: How do researchers prove a specific feature causes a specific output?
A3: They use “clamping” or interventions, where they manually increase or decrease a feature’s activation level and observe if the output changes in a predictable way (e.g., changing the city in a capital city question).

Q4: Can this research help with AI safety?
A4: Absolutely. By identifying “refusal circuits” or “harmful concept circuits,” developers can better understand why a model might bypass safety filters and how to reinforce those filters at a neural level.

Q5: What is the difference between an SAE and a CLT?
A5: A Sparse Autoencoder (SAE) typically looks at a single layer in isolation, whereas a Cross-Layer Transcoder (CLT) is designed to track how features propagate and transform across multiple layers of the model.

Q6: Did the researchers find evidence that models plan their responses?
A6: Yes, specifically in rhyming tasks, where the model activates the “rhyme” feature for the final word of a sentence before it even begins generating that sentence.