28 Neural Network Tricks

The gap between a neural network that works in theory and one that works in practice is often a single trick. A carefully chosen initialisation scheme, a normalization layer inserted in exactly the right place, a learning rate schedule that warms up before it decays, each of these is a small engineering decision, easy to overlook, yet capable of transforming a divergent training run into a state-of-the-art result. The history of deep learning is littered with breakthroughs that were, at their core, tricks: ideas that are easy to implement, hard to discover, and disproportionately impactful.

In 5, we introduced the raw building blocks of deep neural networks (feedforward layers, convolutional layers, residual connections, normalisation layers, and the Transformer's attention mechanism). Those building blocks provide the architecture; the tricks in this chapter provide the engineering wisdom that makes the architecture trainable, stable, and efficient. Without Xavier initialisation, networks deeper than five layers collapse during training. Without residual connections, gradients vanish long before they reach the early layers. Without mixed-precision training, modern LLMs would require twice the memory and twice the time. The tricks are not optional flourishes layered atop a working system; they are the system.

We organise this chapter around fifteen tricks, presented roughly in the order a practitioner encounters them when building a modern neural network. The first five tricks (Trick #1 - Initialize Wisely- Trick #4 - Add Skip Connections Everywhere) address the foundational challenge of getting gradients to flow: how to initialise weights, choose activation functions, normalise intermediate representations, connect layers with gates, and build skip connections. The next five tricks (Trick #6 - Encode Position Cleverly- Trick #10 - Train in Low Precision) tackle the Transformer era's central concerns: encoding position, attending efficiently, regularising creatively, scheduling the learning rate, and training in low precision. The final five tricks (Trick #11 - Distill Knowledge- Trick #15 - Combine Tricks Synergistically) address the deployment challenge of trillion-parameter models: distillation, quantisation, pruning and routing, fast decoding, and the art of combining tricks synergistically.

fig:tricks:timeline provides a serpentine timeline showing when each trick was introduced, revealing how the field has evolved from the foundational ideas of the 1980s, through the Transformer revolution, to the efficiency innovations that dominate current research.

Trick #1 - Initialize Wisely

Weight initialisation is the first decision a practitioner makes when building a neural network, and for decades it was one of the most poorly understood. The consequences of getting it wrong are immediate and catastrophic: activations explode to NaN within a few forward passes, or gradients shrink to machine epsilon within a few backward passes, and the network learns nothing. The consequences of getting it right are equally dramatic: a well-initialised 100-layer network trains as smoothly as a 5-layer one.

Variance Propagation Analysis

The central question of initialisation theory is: how does the variance of activations change as a signal propagates through a deep network? To answer it, we analyse a simple linear network (no activation functions) and then extend the analysis to nonlinear networks with ReLU.

Consider an $L$-layer feedforward network with layer widths $n_0,n_1,\dots ,n_L$. At layer $l$, the pre-activation is (Forward Preact)$$\begin{array}{ccc}& {𝒛}^{(l)}={𝑾}^{(l)}{𝒂}^{(l-1)},& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝑾}^{(l)}\in {\mathrm{ℝ}}^{n_l\times n_{l-1}}$ is the weight matrix and ${𝒂}^{(l-1)}\in {\mathrm{ℝ}}^{n_{l-1}}$ is the activation from the previous layer (with ${𝒂}^{(0)}=𝒙$). We omit biases for clarity; they are typically initialised to zero and do not affect the variance analysis.

Under the assumptions that (i) the weights $W_{ij}^{(l)}\sim 𝒩(0,{\sigma }_l^2)$ are i.i.d., (ii) the weights are independent of the activations, and (iii) the activations have zero mean ($𝔼[a_i^{(l-1)}]=0$), the variance of a single pre-activation component is (VAR Single Layer)$$\begin{array}{ccc}& \mathrm{𝖵ar}\left(z_j^{(l)}\right)=\sum _{i=1}^{n_{l-1}}\mathrm{𝖵ar}\left(W_{ij}^{(l)}\right)\mathrm{𝖵ar}\left(a_i^{(l-1)}\right)=n_{l-1}{\sigma }_l^2\mathrm{𝖵ar}\left(a_i^{(l-1)}\right).& \text{<span class="tml-eqn"></span>}\end{array}$$ This is the fundamental variance recursion: the variance at each layer is scaled by a factor of $n_{l-1}{\sigma }_l^2$ relative to the previous layer.

After $L$ layers, the variance accumulates multiplicatively: (VAR L Layers)$$\begin{array}{ccc}& \mathrm{𝖵ar}\left(z_j^{(L)}\right)=\mathrm{𝖵ar}\left(x_i\right)\prod _{l=1}^L\left(n_{l-1}{\sigma }_l^2\right).& \text{<span class="tml-eqn"></span>}\end{array}$$ Each factor $n_{l-1}{\sigma }_l^2$ acts as a gain: if it exceeds $1$, the variance grows exponentially with depth; if it is less than $1$, the variance shrinks exponentially. Only when $n_{l-1}{\sigma }_l^2=1$ for every layer does the variance remain constant.

We face a tension: the forward pass requires ${\sigma }_l^2=1/n_{l-1}$ (fan-in), while the backward pass requires ${\sigma }_l^2=1/n_l$ (fan-out). These coincide only when $n_{l-1}=n_l$, i.e., all layers have the same width. For networks with varying widths, a compromise is needed.

Xavier/Glorot Initialisation

Glorot and Bengio [2] proposed resolving the fan-in versus fan-out tension by taking the harmonic mean.

Kaiming/He Initialisation

He et al. [3] observed that for ReLU activations, Xavier initialisation causes the variance to halve at each layer, because ReLU zeroes out negative activations. To compensate, they derived a modified variance that accounts for the ReLU's asymmetry.

The key calculation is the second moment of a ReLU-transformed Gaussian. If $z\sim 𝒩(0,{\sigma }^2)$, then $a=\max (0,z)$, and (RELU Second Moment)$$\begin{array}{ccc}& 𝔼[\max (0,z{)}^2]={\int }_0^{\infty }{z}^2\cdot \frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{z^2}{2{\sigma }^2})dz=\frac{{\sigma }^2}{2}=\frac{\mathrm{𝖵ar}(z)}{2}.& \text{<span class="tml-eqn"></span>}\end{array}$$ The integral follows from the half-Gaussian second moment: for a standard Gaussian, $𝔼[Z^2{\mathrm{𝟏}}_{Z>0}]=1/2$, and scaling by ${\sigma }^2$ gives ${\sigma }^2/2$.

Since $𝔼[\max (0,z)]\ne 0$ for the ReLU (the mean is $\sigma /\sqrt{2\pi }$), we have (RELU Variance)$$\begin{array}{ccc}& \mathrm{𝖵ar}(\max (0,z))=𝔼[\max (0,z{)}^2]-(𝔼[\max (0,z)]{)}^2=\frac{{\sigma }^2}{2}-\frac{{\sigma }^2}{2\pi }=\frac{{\sigma }^2}{2}(1-\frac{1}{\pi }).& \text{<span class="tml-eqn"></span>}\end{array}$$ For the purpose of the initialisation derivation, the simpler bound $𝔼[\max {(0,z)}^2]=\mathrm{𝖵ar}(z)/2$ suffices, because we want $\mathrm{𝖵ar}(z^{(l)})\approx \mathrm{𝖵ar}(z^{(l-1)})$ for the pre-activations.

The factor of $2$ compensates for the ReLU halving: each layer's forward-pass gain is $$\begin{array}{ccc}& n_{\mathrm{in}}\cdot {\sigma }^2\cdot \frac{1}{2}=n_{\mathrm{in}}\cdot \frac{2}{n_{\mathrm{in}}}\cdot \frac{1}{2}=1,& \text{<span class="tml-eqn"></span>}\end{array}$$ so the variance of the pre-activations is preserved exactly.

Orthogonal Initialisation

Xavier and He initialisations control the average variance propagation, but they do not guarantee that all singular values of the weight matrix are well-behaved. A Gaussian random matrix has singular values that follow the Marchenko–Pastur distribution, which means some directions may amplify signals while others suppress them. Orthogonal initialisation eliminates this variability entirely.

The key property is dynamical isometry : all singular values of $𝑾$ are exactly $1$. For a product of $L$ orthogonal matrices, ${𝑾}^{(L)}{𝑾}^{(L-1)}\cdots {𝑾}^{(1)}$, all singular values of the product remain exactly $1$ (since the product of orthogonal matrices is orthogonal), so the signal is preserved perfectly, not merely in expectation, but for every input direction.

Saxe et al. [5] proved that orthogonal initialisation produces exact solutions to the learning dynamics of deep linear networks, showing that the training trajectories can be solved in closed form. The gradient flow decomposes along the singular vectors, with each mode converging independently, a remarkable simplification that holds exactly for linear networks and approximately for networks with smooth nonlinearities.

Layer-Sequential Unit-Variance (LSUV)

The methods above derive the initialisation variance from theoretical analysis under simplifying assumptions (linear activations, Gaussian inputs, independence). In practice, nonlinearities, batch normalisation, and complex architectures violate these assumptions. Mishkin and Matas [6] proposed a pragmatic alternative: measure the actual variance empirically and correct it.

The algorithm proceeds layer by layer (sequentially), so that when layer $l$ is calibrated, layers $1,\dots ,l-1$ have already been fixed to produce unit-variance outputs. The orthogonal initialisation in step 2 provides a good starting point; the iterative rescaling corrects for the effects of nonlinearities and architectural peculiarities. In practice, convergence to $|v-1|<0.01$ typically requires only 2–3 iterations per layer.

Fixup Initialisation

For deep residual networks, standard initialisation causes the variance of residual outputs to grow linearly with depth (each residual branch adds variance). Zhang et al. [7] proposed Fixup initialisation: scale the last layer of each residual branch by $L^{-1/(2m-2)}$, where $L$ is the total number of residual blocks and $m$ is the number of layers per block. This factor ensures that the sum of $L$ residual contributions has $O(1)$ variance, eliminating the need for batch normalisation entirely. Fixup was demonstrated on ResNets with up to 110 layers, matching the performance of batch-normalised networks without any normalisation layers.

Maximal Update Parameterisation ($\mu$PmuP)

All the schemes above address a single question: how to start training. Yang and Hu [4] asked a deeper question: how should initialisation, learning rate, and parameterisation co-scale with model width so that the optimal hyperparameters of a small model transfer to a large one?

Standard parameterisation (SP) initialises all layers with the same variance scaling and uses the same learning rate everywhere. Under SP, the optimal learning rate depends on model width: a learning rate that works for a 128-dimensional model is suboptimal for a 1024-dimensional one. This means that hyperparameter tuning must be repeated for every model size, an enormous cost when the target model has billions of parameters.

The name “maximal update” reflects the fact that $\mu$P is the unique parameterisation that (i) ensures stable forward and backward passes and (ii) allows each layer to make the largest possible update without destabilising the network. Under standard parameterisation, hidden-layer updates scale as $O(1/\sqrt{n})$, which shrinks to zero as the model grows, effectively turning large models into “lazy” networks that learn slowly. Under $\mu$P, updates remain $O(1/n)$ in the feature space (the space of activations), which is the correct scaling for feature learning.

Visualising Variance Propagation

fig:tricks:var-propagation illustrates the effect of different initialisation schemes on the variance of activations as a signal propagates through a 10-layer network.

Comparative Summary

Table 1 summarises the initialisation methods discussed in this section.

Method	Activation	${𝝈}^2$	Property	Year
LeCun [1]	$\tanh$	$1/n_{\mathrm{in}}$	Forward-only	1998
Xavier [2]	Linear / $\tanh$	$2/(n_{\mathrm{in}}+n_{\mathrm{out}})$	Fwd + Bwd	2010
He [3]	ReLU	$2/n_{\mathrm{in}}$	ReLU-corrected	2015
Orthogonal [5]	Linear	All ${\sigma }_i=1$	Dyn. isometry	2013
LSUV [6]	Any	Data-driven	Empirical $\mathrm{𝖵ar}=1$	2016
Fixup [7]	ReLU (ResNet)	He + $L^{-1/(2m-2)}$	No norm needed	2019
$\mu$P [4]	Any	Width-dep.	HP transfer	2022

Trick #2 - Make It Gated

Gates are the switches of deep learning. A gate is nothing more than a learned function that outputs values between 0 and 1, multiplied element-wise with a signal to selectively amplify or suppress each dimension. This deceptively simple mechanism (multiply by a number between zero and one) turns out to be arguably the single most important trick in the entire field. Gates appear in recurrent networks, feedforward layers, attention mechanisms, and highway connections. Every major architecture breakthrough of the past decade has either introduced a new kind of gate or applied an existing gate in a new place.

We met gates in the LSTM (Definition 11), where input, forget, and output gates control information flow through the recurrent cell. Here we formalise the general gating mechanism and trace its evolution through the architectures that define modern deep learning.

The General Gating Mechanism

At its core, every gate has the same structure: project the input, apply a sigmoid, and multiply element-wise.

The gate's power comes from its data dependence: the same network dimension may be opened wide for one input and shut for another. This is fundamentally different from a fixed mask or a global scaling factor. The sigmoid activation is the natural choice because its range $[0,1]$ makes it interpretable as a “proportion of signal to let through.”

Highway Networks

Before ResNets, before skip connections became ubiquitous, Srivastava et al. [10] proposed the first architecture to successfully train networks with dozens of layers by using gates to control the information highway.

Gated Recurrent Unit

Cho et al. [9] proposed the GRU as a simpler alternative to the LSTM, reducing three gates to two while retaining comparable modelling power.

The update gate ${𝒛}_t$ plays the role of both the LSTM's input gate and forget gate: when ${𝒛}_t\approx 0$, the hidden state is carried unchanged from the previous step (${𝒉}_t\approx {𝒉}_{t-1}$); when ${𝒛}_t\approx 1$, the hidden state is replaced by the candidate ${\widetilde{𝒉}}_t$. The reset gate ${𝒓}_t$ controls how much of the previous hidden state to expose when computing the candidate: when ${𝒓}_t\approx 0$, the candidate depends only on the current input, effectively “resetting” the memory.

Gated Linear Units and the SwiGLU Family

Dauphin et al. [11] introduced the Gated Linear Unit for language modelling, demonstrating that gating could accelerate convergence even in non-recurrent architectures. The idea is beautifully simple: split the feedforward computation into two parallel paths, one of which passes through a gate.

The key insight is that the gate operates on a linear projection of the input: there is no nonlinearity applied to the value path before gating. This “linear” qualifier is what gives the GLU its name and its gradient-friendly properties: the value path preserves a direct linear pathway for gradients.

Shazeer [12] systematically explored replacing the sigmoid gate with other activation functions, discovering that SiLU-gated variants (SwiGLU) consistently outperformed alternatives. Table 2 summarises the family.

Variant	Gate activation $f$	Introduced by	Used in
GLU	Sigmoid $\sigma (x)$	Dauphin et al. [11]	Original paper
ReGLU	ReLU $\max (0,x)$	Shazeer [12]	-
GEGLU	GELU $x\mathrm{\Phi }(x)$	Shazeer [12]	GPT-J, GPT-NeoX
SwiGLU	SiLU $x\cdot \sigma (x)$	Shazeer [12]	LLaMA, Qwen, DeepSeek

SwiGLU (Definition 20) is now the standard feedforward layer in all major open-weight LLMs, including LLaMA, Mistral, Qwen, and DeepSeek. The SwiGLU FFN ((Swiglu)) replaces the traditional two-layer FFN with a three-matrix computation that includes a gated branch, typically using a hidden dimension of $\frac{8}{3}d$ (rounded to the nearest multiple of 256) to keep the parameter count comparable to the ungated alternative.

Gated Attention: A Deep Dive

Definition 18 defined gated attention; we now examine why it works. Qiu et al. [13] showed that adding a single sigmoid gate to each attention head resolves a fundamental expressiveness bottleneck in multi-head attention and eliminates the infamous “attention sink” phenomenon.

The low-rank bottleneck.

In standard multi-head attention, each head $k$ computes (MHA HEAD)$$\begin{array}{ccc}& {𝒐}_i^{(k)}=(\sum _j{S}_{ij}^{(k)}\cdot {𝒙}_j{𝑾}_V^{(k)}){𝑾}_O^{(k)},& \text{<span class="tml-eqn"></span>}\end{array}$$ where $S_{ij}^{(k)}$ is the softmax-normalised attention weight from token $i$ to token $j$, ${𝑾}_V^{(k)}\in {\mathrm{ℝ}}^{d\times d_k}$ is the value projection, and ${𝑾}_O^{(k)}\in {\mathrm{ℝ}}^{d_k\times d}$ is the output projection. The effective per-head transformation of the value is ${𝑾}_V^{(k)}{𝑾}_O^{(k)}\in {\mathrm{ℝ}}^{d\times d}$, but this matrix has rank at most $d_k=d/h$, where $h$ is the number of heads. For a model with $d=4096$ and $h=32$, each head projects through a rank-128 bottleneck. This limits the expressiveness of each individual head.

Five gating positions.

Qiu et al. [13] systematically evaluated five positions for inserting a gate into the attention computation:

1. G1: Gate after SDPA output. ${𝒀}^{\prime }=𝒀\odot \sigma (𝑿{𝑾}_{\theta })$, where $𝒀$ is the output of scaled dot-product attention. The gate operates on the full $d$-dimensional output, breaking the per-head low-rank constraint.

2. G2: Gate value projection. ${𝑽}^{\prime }=𝑽\odot \sigma (𝑿{𝑾}_{\theta })$, applied before the attention weighted sum.

3. G3: Gate key projection. ${𝑲}^{\prime }=𝑲\odot \sigma (𝑿{𝑾}_{\theta })$, modulating key representations.

4. G4: Gate query projection. ${𝑸}^{\prime }=𝑸\odot \sigma (𝑿{𝑾}_{\theta })$, modulating query representations.

5. G5: Gate output projection. ${𝒐}^{\prime }=𝒐\odot \sigma (𝑿{𝑾}_{\theta })$, applied after the final linear projection.

Of these, G1 performed best, consistently improving perplexity, long-context recall, and training stability. The reason is geometric: G1 is the only position where the gate operates on the full-rank attention output, allowing it to break the per-head rank bottleneck. Gates at positions G2–G5 operate within the low-rank subspace of individual heads and therefore cannot resolve the fundamental expressiveness limitation.

The attention sink problem.

The softmax normalisation forces ${\sum }_j{S}_{ij}=1$: the model must attend somewhere, distributing its full attention budget across the available tokens, even when no token is genuinely relevant to the current query. In practice, this forces the model to concentrate excess attention mass on a convenient “dump” token (typically the first token or the beginning-of-sequence token), creating what Xiao et al. [58] termed an attention sink.

Attention sinks are problematic for three reasons:

1. Wasted capacity: the first token's value vector is mixed into every output, injecting irrelevant information.

2. KV cache dependency: the first token's key and value must be retained forever, complicating sliding-window and streaming inference.

3. Long-context degradation: as the context grows, the sink absorbs an increasingly large fraction of the attention mass, diluting the signal from genuinely relevant tokens.

The gate provides an elegant resolution. When no token is relevant, the gate can simply output ${𝒈}_i\approx 0$, producing a near-zero output without needing to “waste” attention mass on a sink token. The model no longer needs a dump token because it has an alternative mechanism for expressing “nothing here is relevant.”

Empirical results.

The gains from gated attention are substantial and consistent across scales:

• Perplexity: at 15B MoE / 400B tokens, PPL decreased from 6.026 to 5.761 ($-0.265$).

• Long-context recall: on the RULER 128k benchmark, accuracy jumped from 31.65 to 58.82 ($+27.17$ points).

• Training stability: gated attention enabled larger learning rates without training divergence, effectively expanding the feasible hyperparameter space.

• Overhead: the single sigmoid gate adds fewer than $2\%$ to wall-clock training time.

Trick #3 - Normalise Everything

Deep networks are prone to a fundamental instability: as activations propagate through dozens or hundreds of layers, their magnitudes can drift, growing without bound or collapsing to zero. Even when residual connections (Trick #4 - Add Skip Connections Everywhere) prevent gradient death, the activations themselves may still drift to extreme scales, making optimisation difficult. Normalisation layers solve this by enforcing consistent activation statistics at every point in the network.

BatchNorm (Definition 5), LayerNorm (Definition 6), and RMSNorm (Definition 19) were defined in earlier chapters. Here we explore why normalisation works (the answer is more subtle than originally claimed), survey the full family of normalisation methods, and examine the critical question of where to normalise within a Transformer block.

The Covariate Shift Debate

Group Normalisation

BatchNorm's Achilles' heel is its dependence on the batch dimension: the mean and variance are computed across all samples in a mini-batch. When the batch size is small, as in detection and segmentation tasks where high-resolution images consume most of GPU memory, the batch statistics become noisy, and performance degrades sharply. Wu and He [59] proposed Group Normalisation (GN) to eliminate this dependency.

Instance Normalisation

Ulyanov et al. [60] introduced Instance Normalisation for neural style transfer, where removing instance-specific contrast information proved essential for transferring artistic styles.

Comparison of Normalisation Methods

Method	Normalise over	Batch-indep.?	Primary use case
BatchNorm	$(N,H,W)$ per channel	No	Vision (training)
LayerNorm	$(C,H,W)$ per sample	Yes	Transformers
GroupNorm	$(C/G,H,W)$ per group	Yes	Vision (small batch)
InstanceNorm	$(H,W)$ per channel per sample	Yes	Style transfer
RMSNorm	$(C)$ per sample (no mean)	Yes	LLMs

The trend in the field is clear: modern architectures have moved away from batch-dependent normalisation. Transformers use LayerNorm or RMSNorm exclusively, because batch-level statistics are incompatible with autoregressive generation (batch size 1 at inference) and variable-length sequences. RMSNorm ((Rmsnorm)) has emerged as the preferred choice for large language models because it omits the mean subtraction step, saving approximately 10–15% of the normalisation cost while performing comparably [61].

QK-Norm: Normalising Attention Logits

As model dimension $d_{\mathrm{model}}$ grows, the attention logits $𝑸{𝑲}^{\top }$ can grow unboundedly in magnitude, even with the $1/\sqrt{d_k}$ scaling. Henry et al. [62] proposed a simple fix: normalise the queries and keys before computing attention.

DeepNorm: Scaling Normalisation to 1000 Layers

Standard Post-LN Transformers struggle to train beyond approximately 100 layers due to gradient instability. Wang et al. [17] introduced DeepNorm, a modified normalisation scheme that enables stable training of Transformers with up to 1000 layers.

The choice of $\alpha$ and $\beta$ is not arbitrary; it emerges from an analysis of the expected output magnitude across layers.

Pre-Norm vs Post-Norm

The question of where to place normalisation within a Transformer block has been debated since the architecture's inception. The two dominant approaches, and a more recent hybrid, differ in a single design choice that has profound consequences for training dynamics.

Post-norm (original Transformer).

The original Transformer [24] placed LayerNorm after the residual addition: (Postnorm)$$\begin{array}{ccc}& {𝒙}_{l+1}=\mathrm{LN}({𝒙}_l+F({𝒙}_l)).& \text{<span class="tml-eqn"></span>}\end{array}$$ This was noted in Algorithm 2.

Pre-norm (modern standard).

Most modern Transformers place LayerNorm before the sub-layer: (Prenorm)$$\begin{array}{ccc}& {𝒙}_{l+1}={𝒙}_l+F(\mathrm{LN}({𝒙}_l)).& \text{<span class="tml-eqn"></span>}\end{array}$$

Why pre-norm converges faster.

Xiong et al. [63] analysed the gradient flow in both architectures at initialisation and found a crucial difference. In the post-norm configuration, the gradient of the loss with respect to layer $l$ must pass through the LayerNorm Jacobian at every subsequent layer. The LayerNorm Jacobian is a projection matrix that subtracts the mean and rescales; this introduces a contraction at each step, causing gradients to shrink as they propagate backward.

In the pre-norm configuration, the residual path ${𝒙}_{l+1}={𝒙}_l+F(\mathrm{LN}({𝒙}_l))$ provides an unobstructed identity path for gradients. By the chain rule, (Prenorm GRAD)$$\begin{array}{ccc}& \frac{\partial {𝒙}_L}{\partial {𝒙}_l}=𝑰+\sum _{k=l}^{L-1}\frac{\partial F_k(\mathrm{LN}({𝒙}_k))}{\partial {𝒙}_k},& \text{<span class="tml-eqn"></span>}\end{array}$$ where the identity matrix $𝑰$ ensures that the gradient magnitude is at least 1 regardless of depth, analogous to the residual connection's role in gradient preservation.

The quality trade-off.

Despite pre-norm's training advantages, post-norm often achieves better final quality when training succeeds. The reason is subtle: the LayerNorm Jacobian in the post-norm path acts as a form of adaptive regularisation, preventing any single sub-layer from dominating the residual stream. This regularisation effect is lost in pre-norm, where the residual stream is unconstrained and can grow unboundedly, a phenomenon that necessitates the use of a final LayerNorm after the last layer.

Sandwich normalisation.

A natural question arises: can we get the best of both worlds? The Sandwich-LN configuration adds a second LayerNorm after each sub-layer (but before the residual addition): (Sandwich)$$\begin{array}{ccc}& {𝒙}_{l+1}={𝒙}_l+{\mathrm{LN}}_{\mathrm{post}}\left(F\right({\mathrm{LN}}_{\mathrm{pre}}({𝒙}_l)\left)\right).& \text{<span class="tml-eqn"></span>}\end{array}$$ The inner LN (pre-norm) ensures stable input to the sub-layer; the outer LN (post-norm) constrains the sub-layer's output magnitude before it enters the residual stream. This prevents the unbounded growth of the residual stream that plagues pre-norm architectures at very large depth. The cost is two additional normalisation operations per block, a negligible overhead at the scale of modern Transformers.

Trick #4 - Add Skip Connections Everywhere

The basic residual block $𝒚=𝒙+F(𝒙)$ was defined in Definition 9, and the forward pass equation was given in (Residual). In this section we trace the evolution of skip connections from Kaiming He's original insight to the sophisticated multi-stream architectures used in today's largest language models, and explain why the simple act of adding the input back to the output has such a profound effect on trainability.

The Unraveled View of Residual Networks

Veit et al. [64] and Greff et al. [14] offered an elegant reinterpretation: a ResNet is not a single deep computation, but an exponential ensemble of shallow paths.

The critical empirical finding is that most gradient in practice flows through short paths of length 3–5 layers, not through the full-depth path. This explains why ResNets behave more like ensembles of moderately deep networks than like a single very deep one.

Pre-activation Residual Blocks

He et al. [65] identified a subtle but consequential design flaw in the original ResNet: the ReLU after the addition corrupts the identity shortcut.

Why does this matter? The answer lies in the gradient flow.

DenseNet: Concatenation Instead of Addition

Huang et al. [66] pushed skip connections to their logical extreme: connect every layer to every preceding layer.

The crucial difference from ResNet is concatenation instead of addition. Addition merges information destructively: two feature maps are collapsed into one. Concatenation preserves all information, allowing later layers to directly access the raw features from any earlier layer. This direct access path encourages feature reuse: later layers can build on features computed at any earlier stage without recomputing them.

ReZero: Initialise as the Identity

Bachlechner et al. [67] proposed the simplest possible modification to residual connections: multiply the residual branch by a learnable scalar initialised to zero.

At initialisation, ${\alpha }_l=0$ for every layer, so the entire network computes the identity function: ${𝒙}_{l+1}={𝒙}_l$ for all $l$. The signal propagates through the network without distortion, and the gradients flow back without attenuation. As training progresses, the scalars ${\alpha }_l$ gradually grow from zero, allowing the network to learn increasingly nontrivial transformations.

Hyper-Connections: Multi-Stream Residuals

Standard residual connections maintain a single residual stream of dimension $d$. Zhu et al. [68] proposed hyper-connections, which expand the residual stream to $n$ parallel copies and mix them through learnable matrices.

The additional expressiveness comes at a cost: the matrices ${𝑯}_l^{\mathrm{res}}$ are unconstrained. When composed across $L$ layers, their product ${\prod }_{l=1}^L{𝑯}_l^{\mathrm{res}}$ can have spectral norm far exceeding 1. Zhu et al. measured the Amax Gain, the maximum amplification factor of the residual stream, and found values approaching 3000 in a 27B-parameter Mixture-of-Experts model. Such amplification causes severe training instability: loss spikes, divergent gradients, and wasted compute.

Manifold-Constrained Hyper-Connections

Xie et al. [69] proposed a principled solution: constrain the mixing matrices to the Birkhoff polytope, the set of doubly stochastic matrices.

Doubly stochastic matrices appeared in our treatment of optimal transport (Theorem 2), where the Sinkhorn factorisation theorem establishes that any matrix with positive entries can be scaled to a doubly stochastic matrix by alternating row and column normalisation. Here, the same structure provides stability guarantees for deep residual networks.

Property (b) is the crucial one: no matter how many layers we stack, the composed mixing matrix remains doubly stochastic with spectral norm at most 1. This eliminates the exponential amplification that plagued unconstrained hyper-connections.

Sinkhorn–Knopp projection.

In practice, the raw parameters ${\widetilde{𝑯}}_l^{\mathrm{res}}$ are unconstrained. To project onto ${ℬ}_n$, we apply the Sinkhorn–Knopp algorithm (Algorithm 1): (MHC Sinkhorn)$$\begin{array}{ccc}& {𝑴}^{(0)}=\exp \left({\widetilde{𝑯}}_l^{\mathrm{res}}\right),{𝑴}^{(t)}={𝒯}_r\left({𝒯}_c\right({𝑴}^{(t-1)}\left)\right),t=1,\dots ,T,& \text{<span class="tml-eqn"></span>}\end{array}$$ where ${𝒯}_r$ divides each row by its sum and ${𝒯}_c$ divides each column by its sum. The exponential ensures positivity; alternating normalisation converges to a doubly stochastic matrix. In practice, $T=5$ iterations suffice for $n\le 8$.

Trick #5 - Choose Your Activation Wisely

Standard activations (sigmoid, tanh, ReLU, and their variants) were catalogued in Definition 2. Here we examine the subtleties that make one activation preferable to another, from the pathologies of ReLU to the self-normalising properties of SELU to the empirical dominance of GELU and SiLU in modern large language models.

The Dying ReLU Problem

ReLU's hard zero for $x<0$ creates a pathological failure mode: neurons can become permanently inactive.

Under He initialisation with $b=0$, the probability of a neuron dying increases with network depth and learning rate. A large learning rate can push the bias term sufficiently negative that ${𝒘}^{\top }𝒙+b<0$ for all $𝒙$, killing the neuron in a single update step.

SELU: Self-Normalising Networks

Klambauer et al. [20] designed an activation function that makes explicit normalisation layers unnecessary, by ensuring that activations automatically converge to zero mean and unit variance.

The seemingly arbitrary constants $\lambda$ and $\alpha$ are far from arbitrary; they are the unique solution to a pair of integral equations that enforce the self-normalising property.

GELU, SiLU, and the NAS Connection

The two activations that dominate modern large language models are GELU [21] and SiLU (also called Swish) [22]. Recall from (Sigmoid) and (Sigmoid): (GELU Recap)$$\begin{array}{cccc}& \mathrm{GELU}(x)& =x\cdot \mathrm{\Phi }(x),& \text{<span class="tml-eqn"></span>}\\ & \mathrm{SiLU}(x)& =x\cdot \sigma (x),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $\mathrm{\Phi }$ is the standard normal CDF and $\sigma$ is the logistic sigmoid.

These two functions are strikingly similar. The connection is immediate once we note that $\sigma (x)\approx \mathrm{\Phi }(x/1.702)$: the logistic sigmoid is a rescaled approximation to the normal CDF. Substituting: (GELU SILU Relation)$$\begin{array}{ccc}& \mathrm{SiLU}(x)=x\cdot \sigma (x)\approx x\cdot \mathrm{\Phi }(x/1.702)\approx \mathrm{GELU}\left(x/1.702\right)\cdot \mathrm{1.702.}& \text{<span class="tml-eqn"></span>}\end{array}$$ In practice, the difference between GELU and SiLU is negligible for most architectures. GPT-2/3 and BERT use GELU; LLaMA and PaLM use SiLU. The choice is often determined by framework defaults rather than principled comparison.

How Swish was discovered.

Ramachandran et al. [22] used neural architecture search (NAS) to search over the space of activation functions parameterised as $f(x)=x\cdot \sigma (\beta x)$ for learnable $\beta$. The search, conducted across multiple architectures and datasets, consistently converged to $\beta \approx 1$, yielding $f(x)=x\cdot \sigma (x)$, the SiLU function.

Mish

Misra [23] proposed another smooth, non-monotonic activation that shares the beneficial properties of SiLU but uses a different mathematical formulation.

Like SiLU, Mish is smooth, non-monotonic, and unbounded above. It has a slightly deeper negative trough than SiLU (minimum value $\approx -0.31$ at $x\approx -1.19$) and approaches the identity more slowly for large $x$. Empirically, Mish performs comparably to SiLU on most benchmarks, with occasional small advantages on vision tasks (it was the default activation in YOLOv4).

StarReLU: Matching GELU Statistics Cheaply

Yu et al. [70] observed that much of GELU's benefit comes from its output statistics (mean and variance), not its precise functional form. They proposed an activation that matches GELU's statistics using only elementary operations.

StarReLU uses only ReLU and squaring: no exponentials, no error functions, no sigmoid evaluations. On hardware where transcendental functions are expensive (e.g., some mobile accelerators), StarReLU offers a meaningful speedup with minimal accuracy loss. However, on modern GPUs where GELU is implemented as a fused kernel, the computational advantage is negligible, which limits StarReLU's adoption in large-scale training.

Trick #6 - Encode Position Cleverly

Transformers process tokens in parallel, a strength for computation but a weakness for sequence understanding. Unlike recurrent networks, which process tokens one at a time and therefore inherently know the order, a Transformer's self-attention is permutation-equivariant: shuffling the input tokens merely shuffles the output in the same way. Without an explicit position signal, the model cannot distinguish “the cat sat on the mat” from “mat the on sat cat the.”

The sinusoidal positional encoding (Definition 14) and Rotary Position Embeddings (Definition 15), together with the elegant proof that RoPE attention scores depend only on relative position (Proposition 1), were derived in earlier chapters. Here we survey the broader landscape of position encoding strategies, focusing on the practical tricks that determine how far a model can generalise beyond its training context length.

Learned Positional Embeddings

The simplest alternative to sinusoidal encodings is to learn position representations directly from data.

Learned embeddings are maximally flexible: the model can learn arbitrary position-dependent patterns. The fatal limitation is that ${𝑬}_{\mathrm{pos}}$ has exactly $L_{\max }$ rows. At inference time, if the sequence length exceeds $L_{\max }$, there is no embedding for positions $\ge L_{\max }$, and the model cannot generalise.

ALiBi: Attention with Linear Biases

Press et al. [26] proposed an elegantly simple alternative: do not encode position in the embeddings at all. Instead, add a position-dependent bias directly to the attention logits.

There are no learnable parameters associated with the position encoding in ALiBi: the slopes $m_h$ are fixed hyperparameters. Different heads attend at different scales: head 1 (the steepest slope) focuses almost exclusively on nearby tokens, while head $H$ (the shallowest slope) has a nearly uniform attention pattern.

NTK-Aware RoPE Scaling

Given that RoPE (Definition 15) has become the dominant position encoding, the most pressing practical question is: how do we extend the context length of a pretrained RoPE model without full retraining? The simplest approach, position interpolation [71], linearly rescales all positions by the factor $L_{\text{train}}/L_{\text{target}}$. This works but degrades quality on short-range patterns because it compresses the high-frequency rotations.

The key intuition is rooted in the neural tangent kernel (NTK) perspective: the RoPE frequency spectrum spans from very low frequencies (large ${\theta }_i$, capturing long-range structure) to very high frequencies (small ${\theta }_i$, capturing fine-grained local patterns). Position interpolation uniformly compresses all frequencies, destroying the high-frequency information that the model has already learned to use for local token interactions. NTK-aware scaling instead adjusts the base frequency, which effectively stretches the low frequencies more than the high frequencies. The result: long-range positions receive genuinely new frequency representations, while short-range patterns remain largely intact.

YaRN: Yet Another RoPE ExtensioN

Peng et al. [27] observed that neither pure interpolation nor NTK-aware scaling is optimal across the full frequency spectrum. Their solution, YaRN (Yet Another RoPE ExtensioN), partitions the frequency dimensions into three zones and treats each differently.

The logic behind YaRN's three-zone partition is both simple and elegant. Low-frequency RoPE dimensions encode long-range positional information; since the model has never seen positions beyond $L_{\text{train}}$, these dimensions must be interpolated. High-frequency dimensions encode short-range patterns that the model has thoroughly learned; modifying them would destroy this learned structure. Mid-frequency dimensions fall somewhere in between and benefit from a smooth blend.

Position Encoding Comparison

Table 4 summarises the key properties of the position encoding methods discussed in this section and earlier chapters.

Method	Learned?	Extrapolation	Relative?	Key Property
Sinusoidal	No	Limited	No	Fixed; smooth decay with distance
Learned	Yes	None	No	Maximum flexibility; fixed $L_{\max }$
RoPE	No	Moderate	Yes	Rotational; decays with distance
ALiBi	No	Strong	Yes	Exponential decay via attention bias
NTK-RoPE	No	Good	Yes	Frequency-aware base scaling
YaRN	No	Strong	Yes	Three-zone frequency modification

Trick #7 - Attend Efficiently

Self-attention is the computational heart of the Transformer, but it comes with a quadratic price tag. Standard attention (Definition 12) computes $\mathrm{softmax}(𝑸{𝑲}^{\top }/\sqrt{d})𝑽$, requiring $\mathrm{\Theta }(n^{2}d)$ operations and $\mathrm{\Theta }(n^2)$ memory to store the attention matrix. For moderate sequence lengths this is acceptable, but as models push towards million-token contexts, the quadratic cost becomes the dominant bottleneck.

Grouped-query attention (Definition 17), Flash Attention (Algorithm 3) with its IO-complexity analysis (Proposition 2), and multi-head latent attention (Definition 22) were presented in the Transformer chapter. Here we survey additional tricks for taming attention's appetite for compute and memory.

Sparse Attention Patterns

The simplest idea for reducing attention cost is to compute only a subset of the $n^2$ attention entries. Child et al. [72] introduced systematic sparse attention patterns in the Sparse Transformer.

Sliding Window Attention

Sliding window attention takes the local window pattern and makes it the primary attention mechanism, relying on the depth of the network to propagate information across the full sequence.

A crucial insight is that the effective receptive field grows linearly with depth. After $L$ layers of sliding window attention, information from a token at position $j$ can influence a token at position $i$ provided $|i-j|\le L\cdot W$. For a model with $L=32$ layers and $W=4096$, the effective receptive field is $32\times 4096=\mathrm{131,072}$ tokens, sufficient for most practical applications.

Ring Attention

For sequences so long that even the key-value matrices do not fit on a single accelerator, Ring Attention [73] distributes the computation across multiple devices arranged in a ring topology.

Linear Attention

An entirely different approach to reducing attention cost is to approximate the softmax kernel with a decomposable feature map, enabling the computation to be rearranged from $𝒪(n^{2}d)$ to $𝒪(n{d}^2)$.

The key algebraic insight is the associativity trick. In standard attention, we must first compute $𝑨=𝑸{𝑲}^{\top }\in {\mathrm{ℝ}}^{n\times n}$ (cost $𝒪(n^{2}d)$) and then $𝑨𝑽\in {\mathrm{ℝ}}^{n\times d}$ (cost $𝒪(n^{2}d)$). In linear attention, we instead first compute $𝑺=\varphi {(𝑲)}^{\top }𝑽\in {\mathrm{ℝ}}^{d\times d}$ (cost $𝒪(n{d}^2)$) and then $\varphi (𝑸)𝑺\in {\mathrm{ℝ}}^{n\times d}$ (cost $𝒪(n{d}^2)$). The total is $𝒪(n{d}^2)$, which wins whenever $n>d$, a common regime in long-context applications.

Flash Attention Extensions

The core Flash Attention algorithm was presented in Algorithm 3, with its IO complexity analysis in Proposition 2. Here we survey the subsequent extensions that have made Flash Attention the de facto standard for attention computation.

Flash Attention 2.

Dao [74] introduced several improvements over the original:

1. Parallelism over sequence length: the original Flash Attention parallelises over batch size and number of heads. Flash-2 additionally parallelises over the sequence length dimension, improving GPU occupancy for long sequences.

2. Reduced non-matmul FLOPs: by rearranging the online softmax computation, Flash-2 reduces the overhead of non-matrix-multiply operations, which are bottlenecked by GPU register bandwidth.

3. Result: approximately 2$\times$ speedup over the original Flash Attention on A100 GPUs.

Flash Attention 3.

Shah et al. [75] leverage hardware features specific to NVIDIA Hopper GPUs (H100):

1. TMA (Tensor Memory Accelerator): asynchronous data movement between global memory and shared memory.

2. WGMMA (Warpgroup Matrix Multiply-Accumulate): higher-throughput matrix multiply instructions.

3. Warp specialisation: dedicating different warp groups to different tasks (data loading vs. computation) to maximise pipeline utilisation.

4. Result: 1.5–2$\times$ speedup over Flash-2 on H100 GPUs, approaching the theoretical peak throughput.

PagedAttention.

Kwon et al. [76] address a different bottleneck: the KV cache (Definition 16) memory management during inference.

1. Problem: standard KV cache pre-allocates contiguous memory for the maximum sequence length, leading to significant memory fragmentation and waste when serving multiple requests of varying lengths.

2. Solution: PagedAttention manages the KV cache in fixed-size pages (analogous to OS virtual memory), allocating pages on demand and using a page table for indirection.

3. Result: near-zero memory waste, enabling 2–4$\times$ higher serving throughput via better batching.

Trick #8 - Regularize Creatively

Deep networks have enormous capacity, far more parameters than training examples in most practical settings. Classical learning theory would predict catastrophic overfitting. Yet deep networks generalise remarkably well, and a key reason is the regularisation tricks that prevent them from memorising the training data. This section surveys the most important regularisation techniques, unified by a common principle: inject controlled noise during training, and remove it during inference.

Dropout

The scaling factor $1/(1-p)$ in inverted dropout ensures that the expected value is preserved: $𝔼[h_i^{\prime }]=h_i$. This eliminates the need to modify the network at test time, a practical convenience that has made inverted dropout the universal implementation.

DropPath (Stochastic Depth)

Dropout zeros out individual neurons. DropPath, also known as stochastic depth [77], operates at a coarser granularity: it drops entire layers (more precisely, entire residual branches).

The linear decay schedule reflects the empirical observation that later layers in deep networks tend to learn more fine-grained, task-specific features that are more prone to overfitting, while early layers learn general features that should always be active.

Label Smoothing

With hard targets, the cross-entropy loss drives the logits towards $+\infty$ for the correct class and $-\infty$ for all others; the model becomes maximally confident and often overconfident, assigning near-zero probability to all but the correct class. Label smoothing prevents this by ensuring the target distribution always has non-zero probability for every class.

Mixup

CutMix

CutMix offers two advantages over standard Mixup. First, it preserves local spatial structure: within each region, the pixels come from a single coherent image, so the model sees realistic local patterns rather than ghost-like superimpositions. Second, it forces the model to classify from partial information: a training example might show 70% of a dog and 30% of a cat, requiring the model to recognise both from incomplete evidence. This encourages attention to distributed features rather than reliance on a single discriminative region.

R-Drop

Trick #9 - Schedule the Learning Rate

The learning rate $\eta$ in SGD (Definition 4) and its variants is not merely a hyperparameter to be tuned once and fixed. It is a dynamic control signal that shapes the entire training trajectory. Too high, and the optimiser overshoots and diverges. Too low, and it gets trapped in poor local minima or takes unacceptably long to converge. The art of learning rate scheduling is to navigate between these extremes throughout the training process.

Warmup

Why is warmup necessary? At initialisation, the network's representations are essentially random. Gradients computed on these random representations are noisy: they point in roughly the right direction on average, but with high variance. A large learning rate applied to noisy gradients produces large, erratic updates that can push the parameters into regions of the loss landscape from which recovery is difficult or impossible.

Cosine Annealing

The cosine schedule has several appealing properties. It starts at ${\eta }_{\max }$ when $t=0$ and smoothly decays to ${\eta }_{\min }$ when $t=T$, with no abrupt transitions. Crucially, the schedule spends more time at low learning rates: the derivative $d{\eta }_t/dt=-\frac{\pi }{2T}({\eta }_{\max }-{\eta }_{\min })\sin (\pi t/T)$ is smallest near the endpoints, meaning the learning rate changes slowly at the beginning and end. This is desirable because the final phase of training benefits from small, precise updates.

Cosine with warm restarts (SGDR).

Loshchilov and Hutter also proposed periodically restarting the cosine schedule: (SGDR)$$\begin{array}{ccc}& {\eta }_t={\eta }_{\min }+\frac{1}{2}({\eta }_{\max }-{\eta }_{\min })(1+\cos \frac{\pi T_{\mathrm{cur}}}{T_i}),& \text{<span class="tml-eqn"></span>}\end{array}$$ where $T_{\mathrm{cur}}$ resets to $0$ at each restart, and the period can grow geometrically: $T_i=T_0\cdot T_{\mathrm{mult}}^i$ for the $i$-th restart. Each restart allows the optimiser to escape local minima by temporarily increasing the learning rate, then fine-tune into a (potentially better) basin.

Cyclical Learning Rates

Smith [38] made a counterintuitive observation: periodically increasing the learning rate during training can improve both convergence speed and final accuracy.

A key practical contribution from Smith's work is the LR range test, a simple diagnostic for finding the viable range of learning rates for any model and dataset.

One-Cycle Policy

Smith and Topin [78] refined cyclical learning rates into the one-cycle policy, which uses a single cycle of increasing and decreasing learning rate over the entire training run.

WSD: Warmup-Stable-Decay

The most recent addition to the learning rate schedule repertoire is the WSD (Warmup-Stable-Decay) schedule, which was instrumental in training the MiniCPM models [79].

The key insight behind WSD is that the stable phase at constant ${\eta }_{\max }$ serves as a plateau during which the model steadily improves. At any point during the stable phase, one can branch off a copy of the training run and apply a short decay phase to produce a usable checkpoint. This makes the schedule “anytime optimal”: you do not need to commit to a total training budget in advance. With cosine annealing, the learning rate is entangled with the total number of steps $T$, making it impossible to extend training without restarting the schedule.

Trick #10 - Train in Low Precision

Every number in a neural network (every weight, activation, gradient, and optimizer state) is represented as a floating-point number. For decades, the default was 32-bit floating point (FP32), a format designed for scientific computing with $\sim$7 decimal digits of precision. Modern deep learning has shown that this precision is vastly more than needed: networks can be trained with 16-bit or even 8-bit numbers with little or no accuracy loss, while enjoying dramatic speedups and memory savings.

Floating Point Formats

The critical trade-off is between range (determined by the number of exponent bits) and precision (determined by the number of mantissa bits). FP16 has more mantissa bits than BF16 (10 vs. 7), giving it better precision, but FP16's 5-bit exponent limits its range to $\pm \mathrm{65,504}$. BF16 has the same 8-bit exponent as FP32, giving it the same dynamic range ($\pm 3.4\times {10}^{38}$), at the cost of reduced precision.

For training, BF16's range advantage is decisive. Gradient values in deep networks can span many orders of magnitude, and a format that overflows at $\mathrm{65,504}$ (FP16) frequently encounters numerical issues during training of large models. BF16 never overflows in practice, making it far more robust.

Mixed Precision Training

Why keep master weights in FP32? The issue is precision of accumulation. When the learning rate is small (e.g., $\eta ={10}^{-4}$) and the gradient is also small, the update $\eta \cdot g$ may be smaller than the FP16/BF16 precision at the scale of the weight. Concretely, if a weight $w=100$ and the update is $\mathrm{\Delta }w={10}^{-5}$, FP16 cannot represent $100+0.00001\ne 100$ because the mantissa only has 10 bits of precision. In FP32, the 23-bit mantissa resolves this fine difference.

Loss Scaling

Dynamic loss scaling is automatic: the scale $S$ self-adjusts to keep gradients in the representable range of FP16 without requiring any manual tuning. Note that with BF16 training, loss scaling is typically unnecessary because BF16's large dynamic range prevents both overflow and most underflow.

Gradient Clipping

Large gradient norms, caused by outlier examples, loss spikes, or numerical instability, can destabilise training. Gradient clipping constrains the gradient magnitude before the parameter update.

Gradient Accumulation

Modern training often requires large effective batch sizes for stable convergence, but GPU memory may only accommodate a small batch. Gradient accumulation bridges this gap by spreading a single optimiser step across multiple forward-backward passes.

Trick #11 - Distill Knowledge

A large model trained for weeks on thousands of GPUs captures an extraordinary amount of knowledge, not merely the mapping from inputs to correct labels, but the relative confidences across all possible outputs. When a vision model assigns probabilities $[0.7,0.2,0.1]$ to the classes [cat, lynx, dog], the non-zero probability on “lynx” encodes a structural similarity between cats and lynxes that the hard label $[1,0,0]$ entirely discards. Knowledge distillation is the art of transferring this richer signal from a large teacher to a small student.

The distillation objective

The core idea is simple: train the student to match both the true labels and the teacher's soft predictions.

The temperature $T$ controls the “softness” of the probability distribution. At $T=1$, we recover the standard softmax; as $T\to \infty$, the distribution approaches uniform, revealing the full ranking over classes. The factor $T^2$ in (KD LOSS) is not arbitrary; it compensates for the gradient scaling introduced by the temperature.

Self-distillation and Born-Again Networks

A surprising finding: distillation improves performance even when the student has the same architecture as the teacher.

Online distillation

Classical distillation requires a pre-trained teacher. Online distillation removes this requirement by training teacher and student simultaneously.

Distillation at scale

For large language models, distillation takes a different form. The student is typically trained on the teacher's generated text rather than its logit distributions, because logit-level access to frontier models is often unavailable. This is sometimes called black-box distillation: the student learns from the teacher's outputs without access to its internal representations.

Trick #12 - Quantize Aggressively

Modern neural networks are trained in 32-bit or 16-bit floating point, but inference, and increasingly training, can be performed with far fewer bits. Quantization reduces the numerical precision of weights and/or activations, yielding smaller models, faster computation, and lower energy consumption. The challenge is to do so without destroying accuracy.

Post-training quantization

The simplest approach is to quantize a pre-trained model without further training. The key design choice is the mapping from continuous to discrete values.

The quantization error $ϵ=x-\hat{x}$ introduces noise that accumulates through layers. The art of post-training quantization lies in minimising the impact of this accumulated error.

GPTQ: Optimal Brain Quantization

Naïve rounding quantizes each weight independently, ignoring correlations. GPTQ uses second-order information to quantize weights optimally.

AWQ: Activation-Aware Weight Quantization

GPTQ modifies weights after quantization. AWQ takes a complementary approach: it protects important weights before quantization.

QLoRA: Quantize and fine-tune

Quantization is not only for inference. QLoRA enables fine-tuning of quantized models by keeping the base model frozen in low precision and training only small adapter modules.

Trick #13 - Prune and Route

Two complementary ideas address the same question from opposite directions. Pruning asks: “Can we make a trained network smaller without losing accuracy?” Mixture of Experts (MoE) asks: “Can we make a network conditionally larger, using different subsets of parameters for different inputs?” Together, they represent the modern approach to parameter efficiency: use many parameters for capacity, but activate only a few for any given input.

Magnitude pruning

The simplest pruning criterion: remove weights with the smallest magnitudes, on the assumption that small weights contribute least to the output.

The Lottery Ticket Hypothesis

Pruning raises a deeper question: does a dense network need all those parameters, or does it contain a small subnetwork that could have learned just as well from the start?

Mixture of Experts

Rather than making a network smaller, MoE makes it conditionally larger: the total parameter count grows, but each input activates only a small subset of “expert” sub-networks.

A critical challenge in MoE training is expert collapse: the router learns to send most tokens to a few experts, leaving others untrained. The standard remedy is an auxiliary load-balancing loss.

Trick #14 - Cache and Decode Faster

KV caching (Definition 16) and parallel attention+FFN blocks (Definition 21) were covered in the Transformer chapter. Here we turn to techniques that address the fundamental autoregressive bottleneck: generating one token at a time is inherently sequential.

Speculative decoding

The key insight: use a small, fast model to draft several tokens, then verify them all at once with the large model. Because verification of $K$ tokens costs roughly the same as generating one (a single forward pass over $K$ positions), we get a $K\times$ speedup when most drafts are accepted.

The crucial property of speculative decoding is that it produces exactly the same output distribution as the target model.

Medusa decoding

Speculative decoding requires a separate draft model. Medusa eliminates this requirement by adding prediction heads directly to the target model.

Weight tying

A simple but effective trick that reduces parameters and improves performance simultaneously.

Trick #15 - Combine Tricks Synergistically

No trick works in isolation. The power of the techniques covered in Trick #1 - Initialize Wisely–Trick #14 - Cache and Decode Faster lies not in any single trick but in their synergistic combination. A modern large language model is a carefully composed symphony of $10+$ tricks working in concert, each addressing a specific failure mode while remaining compatible with the others.

Model recipe analysis

Table 5 shows which tricks are used by several prominent LLMs. The convergence on a common “recipe” is striking: nearly all models use pre-norm RMSNorm, SwiGLU activations, RoPE positional encoding, and some form of efficient attention.

Trick	LLaMA-2	Qwen-2.5	DeepSeek-V3	Mistral	GPT-4${}^{\ast }$
Init	He	He	He	He	He
Gating	SwiGLU	SwiGLU	SwiGLU	SwiGLU	(unknown)
Norm	RMSNorm	RMSNorm	RMSNorm	RMSNorm	LayerNorm
	(pre)	(pre)	(pre)	(pre)	(pre)
Skip	Residual	Residual	mHC	Residual	Residual
Activation	SiLU	SiLU	SiLU	SiLU	(unknown)
Position	RoPE	RoPE	RoPE	RoPE	(unknown)
Attention	GQA	GQA	MLA	GQA	(unknown)
Regularise	-	-	DropPath	-	(unknown)
LR	Cosine	Cosine	WSD	Cosine	Cosine
Precision	BF16	BF16	BF16+FP8	BF16	(unknown)
MoE	No	No	Yes	No${}^{†}$	Yes
${}^{\ast }$GPT-4 architecture is not publicly disclosed; entries are community estimates.
${}^{†}$Mixtral uses MoE; base Mistral does not.

Interaction effects

Understanding why certain tricks compose well is as important as knowing which ones to use.

The LLaMA recipe: RMSNorm + SwiGLU + RoPE.

These three tricks compose synergistically because each operates on a different aspect of the architecture. RMSNorm is computationally cheap (no mean subtraction, no bias) and stabilises training without introducing extra parameters. SwiGLU adds expressiveness to the FFN block via learnable gating. RoPE handles position without added parameters and naturally supports length extrapolation. None interferes with the others: RMSNorm normalises activations before SwiGLU processes them, and RoPE acts only on attention queries and keys.

Pre-norm + ReZero: redundant?

Pre-norm (Trick #3 - Normalise Everything) already ensures well-behaved gradients at initialisation by normalising inputs to each sub-layer. ReZero's learnable scalar $\alpha$ (initialised at zero) addresses the same problem from a different angle. Combining both is redundant: if $\alpha =0$ at init, the normalisation has nothing to normalise. In practice, one should choose either pre-norm (the standard) or ReZero (simpler but less common), not both.

MoE + quantization challenges.

Different experts may have different weight distributions, making a single quantization scale suboptimal. Per-expert quantization is necessary but increases the number of quantization parameters. Moreover, expert-level load imbalance means that heavily used experts accumulate more gradient noise, potentially requiring expert-specific precision settings.

Gated Attention + Flash Attention.

The element-wise gate $𝒈\odot 𝒐$ in gated attention (Trick #2 - Make It Gated) is applied after the attention computation, so it adds minimal overhead to Flash Attention's IO-optimal algorithm. The two compose cleanly.

Open research directions

1. Automatic trick discovery. Can neural architecture search (NAS) discover new tricks? Most NAS work searches over macro-architecture (layer counts, widths); searching over micro-architecture (normalisation placement, gating variants, skip-connection topology) is largely unexplored.

2. Theoretical understanding. Why do these tricks compose well? Information-theoretic analyses of individual tricks exist (e.g., residual connections preserve mutual information with the input), but a unified theory of trick composition is lacking.

3. Scaling laws for tricks. Do tricks help more or less at scale? Preliminary evidence suggests that some tricks (e.g., careful initialisation) matter less at scale (large models are more robust to initialisation), while others (e.g., MoE, mixed precision) matter more (they enable training at scales that would otherwise be infeasible).

4. Hardware-aware trick design. As hardware evolves (e.g., AMD MI300, Google TPUs, Cerebras wafer-scale), the optimal trick recipe changes. Tricks optimised for NVIDIA A100s may be suboptimal on TPUs (which have different memory hierarchies and matrix unit sizes).

5. Biological analogues. Many tricks have biological counterparts: gating in neurons (ion channel modulation), normalisation in neural populations (divisive normalisation), pruning during development (synaptic pruning in adolescence). Can biological insights suggest new tricks?

Historical Timeline

The techniques covered in this chapter emerged over four decades of research. fig:tricks:full-timeline traces their evolution, from the foundational ideas of the 1980s to the efficiency innovations of the 2020s.

Narrative: four eras of tricks

1986–2010: The dark ages.

Training deep networks was black magic. Backpropagation existed (Rumelhart et al., 1986), but networks deeper than 2–3 layers routinely failed to train. Initialisation was ad hoc (often uniform random), normalisation was non-existent, and the gradient dynamics of deep networks were poorly understood. The few practitioners who achieved results relied on extensive manual tuning and domain-specific architectures.

2010–2015: The foundations.

Three foundational tricks made deep training possible. Xavier initialisation (2010) showed that variance-preserving init enables gradient flow through many layers. Dropout (2012) provided the first effective regulariser for deep networks. Batch normalisation and residual connections (both 2015) made it possible to train networks with $100+$ layers reliably. These tricks transformed deep learning from an art into a science.

2015–2020: The explosion.

The Transformer architecture (2017) [24] combined several tricks (residual connections, layer normalisation, scaled dot-product attention) into a unified framework that scaled to billions of parameters. This era saw rapid innovation: GELU activations, GLU variants, RoPE, mixed-precision training, and the Lottery Ticket Hypothesis. By 2020, the basic recipe for billion-parameter language models was established.

2020–2025: The refinement.

The challenge shifted from “can we train large models?” to “can we train and deploy them efficiently?” Flash Attention made attention IO-optimal. Grouped-query attention reduced KV cache memory. SwiGLU replaced standard FFNs. Quantization (GPTQ, AWQ, QLoRA) made LLMs runnable on consumer hardware. Speculative decoding achieved lossless speedup. RMSNorm replaced LayerNorm. Each refinement was small individually; together, they enabled the deployment of models that would have been impractical in 2020.

Exercises

The following exercises cover all 15 tricks presented in this chapter, ranging from straightforward derivations to open-ended design problems.

The exercises above span the full range of tricks covered in this chapter. Exercises 1–9 test foundational understanding of individual tricks; Exercises 10–16 develop quantitative analysis skills; Exercises 17–23 connect tricks to practical engineering decisions; Exercises 24–28 require numerical computation and verification; and Exercises 29–30 develop design intuition by asking you to analyse interactions and propose new ideas. Solutions to selected exercises (marked with $\star$) are available in Appendix B.