Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization

year: 2017
paper: https://arxiv.org/pdf/1703.06868.pdf
website: https://paperswithcode.com/method/adaptive-instance-normalization
code:
status: skimmed
connections: generative AI, style-transfer, instance normalization

---

Takeaways

Adaptive Instance Normalization

Instance normalization normalizes the input to a single style specified by the affine parameters ($\gamma ,\beta$). $\text{AdaIN}$ was introduced to adapt to arbitrarily given styles (in generative AI) by using adaptive affine transformations. $\text{AdaIN}$ receives a content input $x$ and a style input $y$ and aligns the channel-wise mean and variance of $x$ to match those of $y$. There are no learnable affine parameters (like in $\text{BN},\text{IN }$ or conditional $\text{IN}$). Instead, it “adaptively computes affine parameters from the style input”:

$$\text{AdaIN}(x,y)=y_s\left(\frac{x-\mu (x)}{\sigma (x)}\right)+y_b$$

TLDR.: Style is encoded through scaling and shifting of the values. ($y_s$ and $y_b$ are learnt through a linear layer).

Like in $\text{IN}$ the statistics are computed across spatial dimensions. Intuitively, let us consider a feature channel that detects brushstrokes of a certain style. A style image with this kind of strokes will produce a high average activation for this feature. The output produced by AdaIN will have the same high average activation for this feature, while preserving the spatial structure of the content image. In the original paper, $y_s$ and $y_b$, are replaced by std and mean of the style input, normalizing it to the given style. A bit less flexible

^{4 5}

Link to original