dot product

Dot product

$$a\cdot b=||a||||b||cos(\theta )$$

$\theta$ … angle between $a$ and $b$
equivalent to:

$$a\cdot b=a_1{b}_1+\text{... }+a_n{b}_n$$

The dot product is the unnormalized version of cosine similarity (equivalent up to scale).
It measures how parallel two vectors are. As opposed to the cross product, which measures how perpendicular two vectors are.

The dot product measures the similarity/overlap between two vectors.

Naturally, it shows up everywhere you have similarity-weighted readouts, like associative memory.
It also shows up every time a neuron computes a weighted sum of its inputs $y=w^{T}x$, which leads to the interpretation of neurons as feature detectors, i.e. we can view the neurons weights as a prototype/template the neuron is looking for.

The relation to the cosine and many of the dot product's properties become clear geometrically

“Project” one vector onto another and multiply the length of the other with the resulting vector.
The head of the projected vector follows from drawing an orthogonal line from the head of one vector to another.
Order does not matter.

Dot product is positive if they point into the same direction, negative for opposing directions:

The projected vector $\vec{w}$ has length $0$ if the vectors are perpendicular:

Indifference of order:

→ For two vectors on the unit circle, this projection is the cosine of the angle between them.

Scaling one of those vectors by 2 → dot product is exactly 2x bigger:

$$(2\vec{v})\cdot \vec{w}=2(\vec{v}\cdot \vec{w})$$

Dot product is a type of inner product

Intuition for the cosine formula
khan ac yt

Multiplying the length of $\vec{b}$ times the amount of $\vec{a}$ thats going in the same direction as $\vec{b}$

If they are perfectly parallel, the dot-product is simply $||\vec{a}||||\vec{b}||\cos (90)$, so simply their lengths multiplied!

TODO go through this rq on paper

https://youtu.be/tdwFdzVqito (cosine formula, dot vs cross) 3b1b

See also Matrix Vector Product Intuition