This blog reveals my preliminary thoughts: We should interpret for the future, not the past, which means interpretability should guide/be included in LLM evolution, not just explain what already exists. This includes <1> Discover principles across architectures and <2> Build the interpretability → design loop

Think about this: In 2015, researchers published papers explaining how LSTM works¹. By 2019, those architectures were historical.

Not everything was lost. Some tools endured—probing methods, visualization techniques, the discipline of mechanistic thinking. But the explanations themselves—the intricate stories of how input gates modulate memory, how cell states evolve over time—were tightly bound to an architecture that vanished from the frontier. When LSTM fell out of favor, its interpretability became a museum exhibit: elegant, but inert.

Now consider today. Transformers are everywhere. They power the largest models, dominate benchmarks, and shape how we think about sequence modeling. They are, by all appearances, at their zenith.

And yet—simultaneously—we are publishing papers that catalog their fundamental limitations: quadratic attention costs, poor handling of long-range state, brittle in-context learning, inefficient reasoning pathways. Researchers are already exploring alternatives—state space models, recurrent transformers, mixture-of-experts with dynamic routing, non-autoregressive frameworks. The writing isn’t on the wall yet, but the first draft is being typed.

So here’s the paradox: We are pouring our best interpretability efforts into a system we know will one day be “yesterday”—not because it’s failing now, but because progress demands it be replaced.

If the insights we produce are locked to the Transformer’s wiring—its attention heads, its residual stream, its positional encodings—then what survives when the next paradigm arrives?

Why are we still writing user manuals for a machine whose successor is already being prototyped?

Almost all interpretability work operates under a silent premise:

The current architecture is the object to be understood.

We take the Transformer as given—as a natural system whose internals deserve reverse-engineering. Questions like “What does this attention head compute?” or “How is factual knowledge represented in the residual stream?” assume that the architecture itself is stable, meaningful, and worth explaining on its own terms.

But an architecture is not a phenomenon. It is a hypothesis—a concrete proposal about how intelligence might be implemented. The Transformer hypothesizes that sequence modeling can be solved through token mixing via attention, without explicit recurrence or state. That position can be tested, not just described.

Interpretability, then, should not only dissect how the model works, but ask: Does this architecture actually realize the computational principles it was meant to embody? Yet we rarely frame interpretability this way. Instead of treating the architecture as a testable claim about intelligence, we treat it as the stage on which all understanding must unfold.

This is more than a methodological choice. It reflects a deeper assumption about what counts as explanation—and what gets left out when the stage itself is never questioned.

Those of us who study interpretability care about more than engineering diagnostics. At its best, our work is driven by a deeper question:

How do models think—and what does that reveal about the nature of intelligence itself?

For decades, when people have talked about understanding a system, the dominant approach has been deconstruction: we take a working system, take it apart, and explain how its pieces produce behavior. Interpretability follows this script—model exists, we reverse-engineer it. We ask: "What does this neuron detect?" or "What pattern does this attention head capture?"

This approach is valuable—but inherently reactive and backward-looking. It explains only what is already there, is locked to a specific implementation, and often decays when the architecture is replaced.

But human understanding of intelligence has never been limited to taking things apart. There is another way to understand: through building.

There is a joke picture showing that ‘Theory is when you know everything but nothing work. Practice is when everything works but no one knows why. In [the picture's] lab, theory and practice are combined: nothing works and no one knows why.’ This might show a perspective that people who focus on building or engineering pay no attention to understanding.

Layer 1: Construction as Understanding

Construction itself is a form of understanding—sometimes more powerful than analysis. When we build systems with deliberate variations in training or architecture, we don’t just produce better models; we uncover how intelligence can emerge from specific computational conditions.

Consider RLVR:

When we discover that reward incentives enhance reasoning, we’re not just finding a training trick—we’re uncovering something fundamental about how intelligence emerges from optimization landscapes.

This is not engineering folklore. The “reasoning” in question is not vague competence—it is structured, goal-directed thought that only stabilizes when gradients are shaped by explicit reward signals. By constructing a system where value and representation co-evolve, researchers revealed that reasoning is not an inherent property of scale or architecture, but a behavioral equilibrium sculpted by optimization pressure.

This insight did not come from dissecting a pre-trained model. It arose because we built a system that made the principle visible. No post-hoc probe could have predicted it.

Sometimes, you understand by building. This is understanding through synthesis.

Layer 2: Tools Shape Epistemology

Here is another thing that tools reveal. They reveal what our current tools make thinkable. It gives us a vocabulary for us to imagine what intelligence could be.

The pattern repeats with every architectural breakthrough. Transformers dominate, and suddenly "human thought is autoregressive token prediction." CNNs achieve superhuman vision, and "the visual cortex is just stacked convolutions." RL agents master games, and "the brain is a reward maximizer, dopamine is the gradient."

This is causality running backwards. We didn't build Transformers because we discovered the brain predicts tokens sequentially. We built Transformers because attention was differentiable, parallelizable on GPUs, and happened to scale with data. The architecture succeeded for deeply contingent reasons—hardware constraints, optimization landscapes, engineering convenience. Then we looked at that success and declared: "This must be how intelligence works." The epistemological error is treating engineering solutions as discovered laws. Yes, some insights transfer—hierarchical processing is real in biological vision. But "hierarchical processing exists" is very different from "intelligence is CNNs." One is an empirical observation; the other mistakes a particular implementation for the phenomenon itself.

When we interpret Transformers, we're not decoding universal principles of intelligence. We're explaining how one contingent instantiation—shaped by 2017 and later hardware, optimization techniques, and engineering culture—happens to approximate intelligent behavior. Each architecture doesn't just give us models. It gives us a vocabulary for what intelligence could be—and we forget that vocabulary is contingent, not discovered.

This is not contradictory to Layer 1. It is just a different perspective, not to say these insights are not formal or real ones. It goes further to reveal the "tool-dependent" nature of the insights, which is that tools shape the way we think about intelligence.

Combining all the previous analysis, we should be thinking about a new paradigm, which is still very primary yet challenging. There are some works that trying to interpreting the advanced architectures, or improve model from some insights from interpretability^2–9, but few works or blogs that are very close or relevant.

The new questions become:

can we do interpretability that discovers the non-trivial universalities and counter-intuitive differences across architectures?

And,

can we use these insights to inform the next-generation architecture design, as well as gaining understanding from building?

The new paradigm is Not model-agnostic (I say it is model-blind, for example, SHAP is model-blind) Not the obvious universalities—"models learn features" offers little guidance⁶ or the obvious distinctions (where the insights can be seeked by only checking the architecture design, then you do the interp backwardly). What we need are insights that violate our architectural intuitions (mechanisms that emerge despite not being prescribed) and the differences that matter (show why model A is better on domain D than model B in mechanistic level, for example.)

The new paradigm is Not just applying interpretability insights, but running the closed loop. Simply applying interpretability insights is inefficient and ineffective. Since researchers on interpretability and model architecture just thinks so different! Expect to generate insights that would facilitate engineers is not working at all! they don't care and we don't know how they design architectures!

One of the most significant thing is that we need researchers who can speak both languages—extract mechanistic principles AND implement them architecturally.

The concrete loop looks like:

1. Discover non-obvious mechanisms that explain performance advantages or the differences across architectures.
2. Design the architecture based on discovered mechanisms, improve models or remedy drawbacks.
3. Gain insights and turn it into mechanistic understanding from building the new architecture, then repeat the "discover" process to gain true mechanistic understanding, not just observations.

This has fundamental difference from the old loop (Interpretability as Archaeology):

Old loop (Interpretability as Archaeology): Architecture released → Analyzing → Publish findings → [years later] Architecture already obsolete → Reset with next architecture [though with some transferable legacies]

This isn't about speed—it's about agency. Interpretability becomes a generative force, not a documentary one. We gain predictive power about which architectural changes matter before expensive training runs. We move from "let's try adding another layer" to "this mechanism needs this specific computational structure."

The transformation is fundamental: from passive analysis to active design, from following to leading, from understanding yesterday's models to building tomorrow's.

The work happening now is valuable. Researchers are uncovering real mechanisms—how models compose information, where reasoning emerges, what features enable generalization. These are genuine contributions to understanding intelligence.

But there's a structural bias we need to recognize. Most interpretability research is architecture-specific and deconstructive. We reverse-engineer what's already built, publish insights tightly coupled to current architectures, and move on when the next paradigm arrives. This isn't wrong—it's just incomplete.

The choice facing us is straightforward: we can keep perfecting explanations of soon-to-be-replaced models, or we can build interpretability systems that guide what comes next. We can write obituaries for architectures on their way out, or blueprints for architectures not yet designed.

1. 1

   https://arxiv.org/pdf/1506.02078 ↩

2. 2

   https://arcprize.org/blog/hrm-analysis supercool analysis on The Hidden Drivers of HRM's Performance. Understanding by Engineering. ↩

3. 3

   Attention Sink: discovered and improved. e.g. https://arxiv.org/abs/2309.17453, https://arxiv.org/abs/2505.06708 ↩

4. 4

   Zeyuan Allen-Zhu's CANON http://zeyuan.allen-zhu.com/paper/2025-canon.pdf. The transformer doesn't have horizontal information flow->use CANON to fix it! ↩

5. 5

   H3 https://arxiv.org/pdf/2212.14052 gained motivation from the induction head to have this mechanism included in MAMBA. ↩

6. 6

   https://arxiv.org/abs/2410.06672 universality of features in different architectures. ↩

7. 7

   https://arxiv.org/pdf/2411.12643 DLBACKTRACE: A MODEL AGNOSTIC EXPLAINABILITY FOR ANY DEEP LEARNING MODELS ↩

8. 8

   https://arxiv.org/pdf/2502.01563 When quantization or compression methods preserve massive-value channels, the model’s contextual performance improves significantly compared to baselines that don’t. ↩

9. 9

   https://blog.tilderesearch.com/blog/telescope-stars/ is also a high-level comment. "Mechanistic understanding is the foundation for new architectures, capabilities, and safety tools. Endlessly throwing tricks and compute gets you to benchmarks, but rarely tells you why or how to build on it." ↩

Feedback from @Sharon Li, Oct 16 2025

• It is a good thought, and it needs further thinking to make it more complete.

• Might get stuck in local minima if we only improve architecture based on interpretability, because true evolution might jump far away from the current architecture.

• She suggested I think about whether this paradigm can apply in the history of Artificial Intelligence. For example, can we use this paradigm to invent a better architecture in the RNN era? This is a very insightful comment, because when we are not able to validate it in the future, we can look into the past to find the answer.

• Oct 14 2025 Init Version.
• Oct 27 2025 Update with feedback section and polished the content.