cosine similarity

Cosine similarity is the cosine of the angle between two vectors / the normalized dot product (divided by vector norms).

$$\text{Cosine Similarity}(A,B)=\cos ({\theta }_{AB})=\frac{A\cdot B}{\Vert A\Vert \Vert B\Vert }=\frac{{\sum }_{i=1}^n{A}_i{B}_i}{\sqrt{{\sum }_{i=1}^n{A}_i^2}\sqrt{{\sum }_{i=1}^n{B}_i^2}}$$

$\text{Cosine Similarity}(A,B)\to [-1,1]$
Identical relationships: $[1,2];[2,4]$ → $\theta =0°\to 1$
Orthogonal relationships: $[0,1];[1,0]$ → $\theta =90°\to 0$
Opposite relationships: $[0,1];[0,-1]$ → $\theta =180°\to -1$
The inverse of cosine similarity is called cosine distance and is in the range $[0,1]$.

EXAMPLE

Transform datapoints (e.g. word counts) into a vector.

---

Used in Content-based-addressing for computing attention (ML) scores (similarity of keys and values) for Neural Turing Machine

$$w_t^c(i)=\text{softmax}({\beta }_t\cdot \text{cosine}[{\mathbf{k}}_t,{\mathbf{M}}_t(i)])=\frac{\exp ({\beta }_t\frac{{\mathbf{k}}_t\cdot {\mathbf{M}}_t(i)}{\Vert {\mathbf{k}}_t\Vert \cdot \Vert {\mathbf{M}}_t(i)\Vert })}{{\sum }_{j=1}^N\exp ({\beta }_t\frac{{\mathbf{k}}_t\cdot {\mathbf{M}}_t(j)}{\Vert {\mathbf{k}}_t\Vert \cdot \Vert {\mathbf{M}}_t(j)\Vert })}$$

$\Vert {\mathbf{M}}_t(i)\Vert$

References

Explanation