permutation invariance

A permutation invariant function will give the same output, regardless of the permutation applied to any input.

$$\begin{gathered} f(x):{\mathbb{R}}^n\mapsto {\mathbb{R}}^m \\ f(x[s])=f(x)\text{for any }x\in {\mathbb{R}}^n \\ s:\text{any permutation of indices }\{1,\dots ,n\} \end{gathered}$$

Self-Attention is permutation equivariant, but not permutation invariant

The set transformer makes SA PI by making $Q$ independent of the input.
The sensory neuron

Link to original

References

invariance