Architecture Thread I: Gated Attention, DeltaNet, and Hybrid Models

Paper: Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free¹

The core innovation of this work lies in adding gates to the attention mechanism.

The experimental section is particularly thorough and worth studying. The authors systematically compared 30 gating variants on 15B MoE and 1.7B dense models, arriving at a clear "recommended configuration":

• Position: G₁ (SDPA output)
• Granularity: Both headwise and elementwise work, but elementwise is stronger (though it adds more parameters)
• Head: Head-specific
• Mode: Multiplicative
• Activation: Sigmoid

In other words, simply inserting a head-specific sigmoid gate at the scaled dot-product attention (SDPA) output position can stably improve performance. The benefits include:

• Perplexity drops significantly (up to 0.2)
• MMLU improves by ~2 points
• Training becomes more stable (loss spikes disappear), enabling larger learning rates
• Better scaling laws

The analysis section contains several interesting insights:

1. Gate as Nonlinearity Map

Adding a gate is equivalent to inserting a nonlinearity map at a specific position. For example, G₁ can be expressed as:

$$o_i^k = \text{NonLinearityMap}\left(\sum_{j=0}^i S_{ij}^k \cdot X_j W_V^k\right) W_O^k$$

Using similar reasoning, we can understand why G₅ doesn't work: the nonlinearity map is added outside, failing to break the low-rank property of $W_V W_O$.

But this doesn't explain why G₁ performs better than earlier positions (G₂, G₃, G₄). My intuition when reading was that G₁, being after SDPA output, has already completed context aggregation (which earlier gates cannot achieve). The paper's explanation is: when gating scores are determined by the current query, their sparsity is more effective; if determined by key/value, performance degrades. Specifically, G₁'s gating scores come from the hidden state corresponding to the current query, while G₂'s gating scores come from hidden states corresponding to past key/values:

$$o_i^k = \left(\sum_{j=0}^i S_{ij}^k \cdot \text{NonLinearityMap}(X_j W_V^k)\right) W_O^k$$

This means the sparsity of gating scores can filter out contextual information irrelevant to the current query.

2. Conclusions About Sparsity

Effective gating scores are sparse. Head-specific sparsity is crucial. When they replaced sigmoid with:

$$\text{NS-sigmoid}(x) = 0.5 + 0.5 \cdot \sigma(x)$$

performance degraded significantly compared to original sigmoid, indicating that both nonlinearity and sparsity are essential—neither alone suffices.

3. Attention Sink Elimination

Intuitively, softmax cannot output "I don't care" zero vectors, so attention must be distributed to some token, making the first token a default garbage bin. Gates give attention heads the ability to "shut up" for the first time—when a head finds a context segment useless, it can directly compress the entire head output to zero, no longer dumping attention into the garbage token. The sink, no longer forced to receive traffic, naturally disappears.

Papers:

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length²
• Gated Delta Networks: Improving Mamba2 with Delta Rule³

Blog: DeltaNet Explained (Part I, II, III)⁴

The Delta Rule is a remarkably simple idea rooted in "error-driven" learning. DeltaNet applies this principle to Linear Attention, transforming the original linear attention's "blind write":

$$S_t = S_{t-1} + v_t k_t^\top$$

into an online memory update with error signals:

$$S_t = S_{t-1} - \beta_t (S_{t-1}k_t - v_t) k_t^\top$$

This equation's meaning is straightforward:

• $S_{t-1}k_t$: Old memory's prediction for key_t
• $v_t$: True value
• Error = $v_t - S_{t-1}k_t$
• $\beta_t$: Controls the strength of memory update

Defining $v_t^{\text{old}} = S_{t-1}k_t$ and $v_t^{\text{new}} = v_t$, the same update expands to:

$$S_t = S_{t-1} - \beta_t v_t^{\text{old}} k_t^\top + \beta_t v_t^{\text{new}} k_t^\top$$

This corresponds to two steps:

1. Erase: Subtract $\beta_t v_t^{\text{old}} k_t^\top$, the scaled old prediction's contribution
2. Write: Add $\beta_t v_t^{\text{new}} k_t^\top$, the scaled new value's contribution

Coincidentally, DeltaNet's update rule can be derived by minimizing the mean squared error (MSE) between expected and predicted outputs at each timestep. For:

$$\mathcal{L}_t(S) = \frac{1}{2}\|Sk_t - v_t\|^2$$

Applying gradient descent to minimize the MSE loss yields:

$$\begin{align} S_t &= S_{t-1} - \eta_t \nabla \mathcal{L}_t(S_{t-1}) \\ &= S_{t-1} - \eta_t (S_{t-1}k_t - v_t) k_t^\top \end{align}$$

This explains why DeltaNet excels at in-context retrieval.

The above content is still not difficult. The original DeltaNet treated it as a pure RNN, which is inefficient on modern GPUs. Therefore, we need strategies to parallelize DeltaNet across sequence length for hardware-efficient training. However, this content is lengthy, and I plan to fully update it in the next post.

Modern Architecture Improvements

A DeltaNet block essentially mimics modern Transformers—alternating token-mixing (using Delta instead of self-attention) and channel-mixing (e.g., SwiGLU). Improvements include:

1. Normalization and Activation

Replace original L₁ normalization and 1+ELU activation with L₂ normalization and SiLU activation for query and key processing.

DeltaNet's update rule is:

$$S_t = S_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$$

To prevent this system from exploding, the eigenvalues of matrix $I - \beta_t k_t k_t^\top$ must be controlled within a reasonable range. Its structure is clean: in the $k_t$ direction, the eigenvalue is $1 - \beta_t \|k_t\|^2$; in all other directions, eigenvalues are $1$. Thus, keeping $\|k_t\|$ stable stabilizes the entire state update. L₂ normalization ensures $\|k_t\| = 1$, and since $0 < \beta_t < 1$, we have $0 < 1 - \beta_t \|k_t\|^2 < 1$, guaranteeing eigenvalues won't explode.

2. Short Convolution

After linear projections for query, key, and value, add short convolution. This provides a shortcut for forming induction heads, enhancing in-context learning capability.

3. Output Normalization

Before the final projection, we normalize the output. Instead of softmax-like denominator normalization, use output-side normalization (RMSNorm). In linear attention, denominator normalization can easily cause gradient explosion.

The improved module structure:

• Query/Key: Linear → Short Conv → SiLU → L₂ Norm
• Value: Linear → Short Conv → SiLU
• Beta: Linear → Sigmoid
• Output: Delta rule(Q, K, V, β) → RMSNorm → Linear

Gated DeltaNet

Gated DeltaNet further improves upon DeltaNet:

$$S_t = S_{t-1} (\alpha_t (I - \beta_t k_t k_t^\top)) + \beta_t v_t k_t^\top$$

By introducing an adaptive scaling factor $\alpha_t$, Gated DeltaNet allows controlled deviation between $S_t$ and $S_{t-1}$, solving the "state saturation" problem.

Similarly, Gated DeltaNet's hardware-efficient implementation is planned for the next post's complete update.

Blog: Qwen3-Next: Towards Ultimate Training & Inference Efficiency⁵

Qwen3-Next demonstrates a rare, genuinely architecture-level evolution. Rather than stacking local modifications on the traditional Transformer framework, it validates the feasibility and reliability of entirely new structures at industrial scale. Most similar attempts typically remain in research papers, while the Qwen team chose to deploy directly in large-scale training.

Qwen3-Next's design core focuses on two points:

1. Hybrid Architecture

Gated DeltaNet and Gated Attention combined in a 3:1 ratio, consistently outperforming single architectures across different scales, balancing model capability and inference efficiency.

2. Extreme Sparsity MoE

Activates only ~3.7% of parameters, achieving more stable and scalable sparse training through global expert balancing mechanisms.

Both designs correspond to the team's recent research achievements:

Paper: Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free¹

Paper: Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models⁶

Overall, such aggressive structural exploration has been uncommon in previous large model development. Qwen3-Next's successful deployment demonstrates that future model design pathways to break through performance bottlenecks come not only from scaling, but more from proactive innovation at the architectural level.

The innovations presented here—Gated Attention, DeltaNet, and their hybrid implementations—represent fundamental rethinking of attention mechanisms beyond traditional Transformers. The success of Qwen3-Next in production validates that architectural innovation, not just scale, is crucial for advancing the field. These developments open new directions for efficient, capable language models that balance computational efficiency with modeling power.

1. 1

   Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free https://arxiv.org/abs/2505.06708 ↩

2. 2

   Parallelizing Linear Transformers with the Delta Rule over Sequence Length https://arxiv.org/abs/2406.06484 ↩

3. 3

   Gated Delta Networks: Improving Mamba2 with Delta Rule https://arxiv.org/abs/2412.06464 ↩

4. 4

   DeltaNet Explained (Part I, II, III) https://sustcsonglin.github.io/blog/2024/ ↩

5. 5

   Qwen3-Next: Towards Ultimate Training & Inference Efficiency https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list ↩

6. 6

   Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models https://arxiv.org/abs/2501.11873 ↩

I've recently started paying closer attention to work on LLM model architectures—both Attention-based improvements and architectural innovations beyond Transformer. This direction has turned out to be more fascinating than I initially expected.

A few days ago, I made a decision: I can't just keep reading. I need to start writing down my thoughts and notes. Partly to document and push myself to learn, partly to force myself to refine my understanding. So today, I'm officially starting.

I began with Qwen3-Next's Hybrid architecture. Following the thread naturally led me recursively into a series of works: Gated Attention, DeltaNet, and Gated DeltaNet. Along the way, I found myself branching out to fill in foundational knowledge in this area.

As I read, a strong feeling emerged:

Modern Transformers have evolved significantly compared to the 2017 "Attention is All You Need" paper. Many components have gone through multiple iterations and refinements, and the exploration continues.

At the same time, more and more researchers are trying to answer a more fundamental question: Can we create an entirely new architecture that truly surpasses Transformer? Even at Transformer's peak, these challengers keep emerging. These new architectures aren't perfect yet—some still underperform compared to well-tuned Transformers on certain tasks—but they represent something more important: exploration of paradigm boundaries, a rethinking of "what makes an architecture truly good."

Transformer's success likely lies in answering the right question for 2017. But that doesn't mean it will answer the right question for 2027. The value of these new improvements and architectures isn't just in current performance—it's in opening up the design space, showing us that every exploration of a new architecture helps us understand more deeply "what actually makes an architecture work." And that understanding is precisely the necessary path to the next paradigm.

This is my first time organizing content like this, and my experience is still limited. If there are any inaccuracies or imprecise understandings, I very much welcome corrections. I'll continue going deeper in this direction, documenting the structures, aesthetics, and mechanisms I encounter along the way.