10 Wasserstein GAN

The WGAN critic has no sigmoid output layer. Its output is an unbounded real number (a “score”), not a probability. Using a sigmoid would destroy the Wasserstein interpretation and revert to something closer to the original GAN.

Weight clipping is a blunt instrument:

• Capacity underuse. If $c$ is too small, the critic can only represent very simple functions, wasting the network's capacity. In practice, clipped critics tend to learn functions that are nearly linear.

• Exploding/vanishing gradients. If $c$ is too large, the Lipschitz constant becomes very large and the Wasserstein estimate is poor; if $c$ is too small, gradients can vanish through many layers.

• Sensitivity to $c$. Performance depends critically on the clipping range, and the optimal $c$ varies across datasets and architectures.

WGAN-GP is incompatible with batch normalisation in the critic. Batch normalisation introduces dependencies between samples in a mini-batch, which means the critic's gradient at $\widehat{𝒙}$ depends on other samples in the batch. This violates the assumption that the gradient penalty is evaluated independently for each interpolated point. Use layer normalisation or instance normalisation instead.

Let $f:{\mathrm{ℝ}}^n\to {\mathrm{ℝ}}^m$ be $L_1$-Lipschitz and $g:{\mathrm{ℝ}}^m\to {\mathrm{ℝ}}^p$ be $L_2$-Lipschitz. Show that $g\circ f$ is $L_1{L}_2$-Lipschitz. Conclude that a neural network with $\ell$ layers, each with Lipschitz constant $L_i$, has overall Lipschitz constant at most ${\prod }_{i=1}^{\ell }{L}_i$.

Consider a single-layer linear critic $f_w(𝒙)={𝒘}^{\top }𝒙$ with $𝒘\in {[-c,c]}^d$.

1. (a)

   Show that the Lipschitz constant of $f_w$ is ${\Vert 𝒘\Vert }_2$.

2. (b)

   Show that weight clipping enforces ${\Vert 𝒘\Vert }_2\le c\sqrt{d}$, not ${\Vert 𝒘\Vert }_2\le 1$.

3. (c)

   Explain how this gap affects the Wasserstein estimate.

Starting from Proposition Proposition 2, explain why:

1. (a)

   The gradient penalty is evaluated at interpolated points rather than at real or fake points alone.

2. (b)

   The two-sided penalty ${({\Vert \nabla f\Vert }_2-1)}^2$ is preferred over the one-sided penalty $\max {(0,{\Vert \nabla f\Vert }_2-1)}^2$.

3. (c)

   The penalty coefficient $\lambda$ should be large enough to enforce the constraint but not so large that it dominates the Wasserstein objective.

Let $p_{\mathrm{data}}={\delta }_0$ and $p_{\theta }={\delta }_{\theta }$ on $\mathrm{ℝ}$.

1. (a)

   Compute $W_1(p_{\theta },p_{\mathrm{data}})$.

2. (b)

   Compute ${𝖣}_{\mathrm{JS}}(p_{\theta }\Vert p_{\mathrm{data}})$.

3. (c)

   Compute the gradients ${\nabla }_{\theta }{W}_1$ and ${\nabla }_{\theta }{𝖣}_{\mathrm{JS}}$ for $\theta \ne 0$.

4. (d)

   Explain why this illustrates the advantage of WGAN for training generative models.

For distributions $P,Q$ on $\mathrm{ℝ}$ with CDFs $F_P,F_Q$:

1. (a)

   Show that $W_1(P,Q)={\int }_{-\infty }^{\infty }|F_P(x)-F_Q(x)|\mathrm{d}x$. Hint: the optimal transport maps quantiles to quantiles.

2. (b)

   Compute $W_1(𝒩(\mathrm{0,1}),𝒩(\mu ,1))$ and show it equals $|\mu |$.

3. (c)

   Compute $W_1(𝒩(0,{\sigma }_1^2),𝒩(0,{\sigma }_2^2))$.

Let $𝒗$ be a unit vector in ${\mathrm{ℝ}}^d$ and $𝒙$ an arbitrary vector.

1. (a)

   Show that $𝒗$ and $𝒙-({𝒙}^{\top }𝒗)𝒗$ are orthogonal.

2. (b)

   Prove ${\Vert 𝒙\Vert }^2={({𝒙}^{\top }𝒗)}^2+{\Vert 𝒙-({𝒙}^{\top }𝒗)𝒗\Vert }^2$.

3. (c)

   Use this decomposition to verify the squeeze argument in the proof of Proposition Proposition 2.

Let $k(𝒙)={\inf }_{𝒖}[\Vert 𝒙-𝒖\Vert -g(𝒖)]$ as in the KR duality proof.

1. (a)

   Show that if $f(𝒙)+g(𝒚)\le \Vert 𝒙-𝒚\Vert$ for all $𝒙,𝒚$, then $f(𝒙)\le k(𝒙)$.

2. (b)

   Show $-k(𝒚)\ge g(𝒚)$ for all $𝒚$.

3. (c)

   Combine (a) and (b) to prove .

For the matrix $W=\left(\begin{array}{cc}2& 1\\ 0& 3\end{array}\right)$:

1. (a)

   Compute the singular values of $W$.

2. (b)

   Compute $\bar{W}=W/\sigma (W)$.

3. (c)

   Verify that ${\Vert \bar{W}\Vert }_2=1$.

4. (d)

   Perform two steps of power iteration starting from ${𝒖}_0={(\mathrm{1,0})}^{\top }$ and compare with the exact answer.

1. (a)

   Show that ReLU is $1$-Lipschitz.

2. (b)

   Show that LeakyReLU with slope $\alpha$ is $\max (1,|\alpha |)$-Lipschitz.

3. (c)

   For a two-layer network $f(𝒙)=W_2\mathrm{ReLU}(W_1𝒙)$, show that ${\Vert f\Vert }_L\le {\sigma }_{\max }(W_2){\sigma }_{\max }(W_1)$.

4. (d)

   Is this bound tight? Give an example.

Consider training a convolutional GAN on CIFAR-10.

1. (a)

   Describe the critic architecture for each of WGAN (weight clipping), WGAN-GP, and SN-GAN. Which normalisation layers can each use?

2. (b)

   If the critic has 5 convolutional layers each with $3\times 3$ kernels, estimate the computational overhead of the gradient penalty versus spectral normalisation.

3. (c)

   Why does WGAN-GP require Adam with ${\beta }_1=0$ while standard GAN training uses ${\beta }_1=0.5$?

In the original GAN, the optimal discriminator computes the density ratio $p_{\mathrm{data}}/(p_{\mathrm{data}}+p_{\theta })$ (Proposition Proposition 1). What does the optimal WGAN critic compute? Show that it is related to the Kantorovich potential and not a density ratio.

The Fréchet Inception Distance (FID, Definition Definition 7) is $$\mathrm{FID}={\Vert {𝝁}_r-{𝝁}_g\Vert }^2+\mathrm{trace}({\mathrm{\Sigma }}_r+{\mathrm{\Sigma }}_g-2({\mathrm{\Sigma }}_r^{1/2}{\mathrm{\Sigma }}_g{\mathrm{\Sigma }}_r^{1/2}{)}^{1/2}).$$

1. (a)

   Show that this equals $W_2^2(𝒩({𝝁}_r,{\mathrm{\Sigma }}_r),𝒩({𝝁}_g,{\mathrm{\Sigma }}_g))$.

2. (b)

   Explain why using $W_2$ in Inception feature space rather than pixel space is advantageous.

Suppose $P$ and $Q$ are distributions over a finite set $𝒳=\{1,2,\dots ,n\}$ with metric $d(i,j)=|i-j|$.

1. (a)

   Write $W_1(P,Q)$ as a linear program.

2. (b)

   Show that the KR dual specialises to $W_1={\max }_{|f(i)-f(j)|\le |i-j|}{\sum }_{i}f(i)(P(i)-Q(i))$.

3. (c)

   For $P=(0.5,0.5,0)$ and $Q=(0,0.5,0.5)$, compute $W_1$ and find the optimal $f$.

Write PyTorch pseudocode for:

1. (a)

   The gradient penalty computation, using torch.autograd.grad to compute ${\nabla }_{\widehat{𝒙}}{f}_w(\widehat{𝒙})$.

2. (b)

   A training loop that alternates between critic and generator updates.

3. (c)

   Explain why create_graph=True must be passed to torch.autograd.grad for the gradient penalty.