multi-layer perceptron

An MLP is a feedforward neural network consisting of multiple fully connected layers, with nonlinear activation functions in between.

They are usually applied position wise to input features for sequential data (especially if the sequence length varies and sliding window doesn’t make sense).

MLP block with residual connection

$$\text{MLPBlock}=\varphi (x{W}_{\uparrow })W_{\downarrow }+x$$

$x\in {\mathbb{R}}^{1\times d}$
$W_{\uparrow }\in {\mathbb{R}}^{d\times k\cdot d}$
$W_{\downarrow }\in {\mathbb{R}}^{k\cdot d\times d}$
$k$ … expansion factor. Usually $k=4$

transformer-feed-forward-layers-are-key-value-memories

$W_{\uparrow }$ … kd key vectors
$W_{\downarrow }$ … kd value vectors
$\varphi$ keeps only the strong matches