📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=IHikLL8ULa4

Decoding the Ghost in the Machine: How Models “Grok” Through Trigonometry

For months, researchers were baffled by neural networks that suddenly learned to generalize after long periods of stagnation. This deep dive into mechanistic interpretability reveals the hidden circuits that turn brute-force memorization into elegant mathematical algorithms.

Core Question: What internal structural changes occur within a neural network when it transitions from rote memorization to true algorithmic generalization?

Highlights

• Grokking is a “cleanup” phase where generalizing circuits eventually outperform complex memorization strategies.
• Small Transformers solve modular addition by mapping numbers to rotations on a unit circle rather than performing standard arithmetic.
• “Progress measures” like Restricted Loss can detect hidden learning long before the test accuracy shows any improvement.
• Regularization, specifically weight decay, acts as the crucial pressure that forces models to abandon complex memorization for simpler logic.

⏱️ Reading time: approx. 7 minutes · Saves you about 36 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Mystery of the Sudden Leap

The Plateau and the Spike

Grokking represents one of the most counterintuitive phenomena in deep learning research today. A model is trained on a tiny dataset, hits 100% training accuracy almost instantly, and then sits in a plateau for thousands of steps before suddenly “getting it” and generalizing to new data.

It is a delayed moment of clarity that challenges our current understanding of how gradients navigate complex, high-dimensional loss landscapes.

Unlike double descent, which usually varies with model size or data volume, grokking happens over time on the exact same data. The model sees the same numbers over and over, yet something internal shifts fundamentally. Neel Nanda and his team argue that this isn’t a random stumble into a solution but a structured competition between two internal strategies: memorization and generalization.

💡 Digging Deeper

Q: Is grokking just another name for double descent?
A: Not exactly. Grokking happens as a function of training time on the same fixed dataset, whereas double descent is usually linked to model size or training data volume.

Q: Why is weight decay so critical for this to happen?
A: Weight decay acts as a simplicity bias, constantly pushing the model to reduce its weight norm, which eventually makes high-norm memorization unsustainable compared to low-norm generalization.

The Trigonometric Circuit

Rotation Over Arithmetic

When the researchers looked inside a one-layer Transformer trained on modular addition, they found something “galaxy-brained” that they hadn’t programmed. Instead of performing standard addition, the model maps each input number to a specific rotation on a unit circle. To add A and B, the model simply composes these two rotations. This naturally handles the “modulo” aspect because a full rotation brings you right back to the start, canceling out any overflow automatically.

The network essentially reinvented basic trigonometry to solve a discrete math problem, utilizing phase rotations to bypass the messiness of standard carry-the-one arithmetic.

By analyzing the output logits, the team discovered that the model creates a probability distribution by rotating the internal representation backward for every possible answer. If the backward rotation lands on the correct coordinate, the x-axis value is maximized, signaling the right answer to the softmax layer. It is a clean, interpretable, and surprisingly geometric solution hidden within a sea of seemingly random weights.

💡 Digging Deeper

Q: Does the model only use one circle?
A: No, it actually learns multiple frequencies or “rotations” at different speeds, which provides redundancy and helps the model pinpoint the exact modular result.

Q: How do you “see” a rotation in weights?
A: By using Fourier transforms on the model’s weight matrices, researchers can identify the specific frequencies the model is using to represent the numbers.

Measuring the Invisible

Identifying the Three Phases

The breakthrough of this paper lies in the development of “progress measures” that make the invisible visible. Usually, we only see the final accuracy spike, which makes the learning process look like a discrete, magical jump. By developing metrics like “Restricted Loss,” researchers can isolate the generalizing circuit and see it growing steadily even while the overall model performance looks stagnant.

Generalization isn’t a happy accident; it is a slow-motion takeover of the model’s internal processing power.

These measures reveal three distinct phases of training. First, the model uses its high capacity to simply memorize the training set, which is the easiest path to low loss. Next, a circuit formation phase begins where the model starts building the trigonometric logic in the background, though it is not yet strong enough to influence the output. Finally, a “cleanup” phase occurs where the weight decay penalizes the heavy memorization weights, forcing the model to rely solely on the leaner, generalizing circuit.

The Mechanics of Learning

Scaling and Future Forecasting

We shouldn’t assume every large-scale model groks in this specific, clean way, as modular addition is a very specific type of problem.

The study focuses on toy models and modular arithmetic, which are much “cleaner” than real-world text datasets. Large language models like GPT-4 rarely encounter the same data enough times to exhibit this exact behavior, as they are usually trained for only one or two epochs. However, the underlying principle—that models develop internal sub-circuits that compete for dominance—is likely a fundamental truth of how all neural networks function.

One of the biggest hurdles remains the predictive power of these interpretability tools. While we can now explain why a model grokked after the fact, we still struggle to forecast the exact moment the “lightbulb” will turn on in a new task. The ultimate goal is to eventually detect deceptive or dangerous circuits in AGI before they become active, moving mechanistic interpretability from a post-mortem tool to a preventative safety measure.

Key Takeaways

Mechanistic interpretability demonstrates that what appears to be a sudden “leap” in machine intelligence is actually the result of gradual, hidden structural changes. By looking at the weights not as random numbers but as components of an algorithm, we can see the model transitioning from rote memorization to a more efficient, trigonometric solution.

The discovery of the cleanup phase is particularly significant for AI safety. It suggests that models may harbor “hidden” capabilities or circuits that are functional but suppressed by more dominant, less efficient strategies. Understanding how to measure these latent circuits could be the key to predicting when a model will suddenly acquire a new, potentially dangerous skill.

Ultimately, grokking is a product of the interaction between the complexity of the task, the amount of data available, and the regularization pressures applied. When the generalizing solution is simpler than memorizing every individual data point, the model will eventually find it, provided the training is long enough and the “simplicity bias” is strong enough.

Q&A

Q1: What exactly is “Mechanistic Interpretability”?
A1: It is a subfield of AI research focused on reverse-engineering neural networks to understand the specific algorithms and circuits they have learned, similar to how one might reverse-engineer a computer chip.

Q2: Why does the model choose trigonometry for addition?
A2: Because modular addition is periodic, and trigonometric functions (sines and cosines) are the natural mathematical language for periodicity and rotation.

Q3: Can we use these progress measures to predict when a model will finish training?
A3: Not perfectly yet. While they show the model is making progress, we cannot yet pinpoint the exact training step where the accuracy will spike.

Q4: Does grokking happen without weight decay?
A4: Generally no. Without a penalty on weight size, the model has no incentive to “clean up” its messy, high-norm memorization circuit even after the generalizing one is formed.

Q5: What is “Restricted Loss”?
A5: It is a metric created by the researchers where they manually delete the parts of the model responsible for memorization to see how well the remaining “generalizing” parts perform.

Q6: Is this research applicable to Large Language Models (LLMs)?
A6: It provides a proof of concept. While LLMs are too complex to map fully today, the concepts of circuit formation and competition are expected to scale to larger architectures.

Q7: What is the “Adam Optimizer” digression about?
A7: The researchers found that the default settings of the Adam optimizer can actually hide or delay grokking because it “remembers” gradients from the early memorization phase for too long.