permutation invariance

A permutation invariant function will give the same output, regardless of the permutation applied to any input.

$$\begin{gathered} f(x):{\mathbb{R}}^n\mapsto {\mathbb{R}}^m \\ f(x[s])=f(x)\text{for any }x\in {\mathbb{R}}^n \\ s:\text{any permutation of indices }\{1,\dots ,n\} \end{gathered}$$

Self-Attention is permutation invariant and permutation equivariant.

NOTE: The full transformer is not - set transformer changes this.
(from set transformer)

Link to original

References

invariance