eigendecomposition

Diagonalization: Performing a transformation from the perspective of the eigenbasis.

We can exploit the efficiency of diagonal matrices for taking large powers if we convert a transformation matrix $A$ into a diagonal one by multiplying it with the change-of-basis matrix $S$ and its inverse:

$$D=S^{-1}AS$$

“First translate from the language of the eigenbasis to our basis, then apply $A$, then translate back. The resulting matrix $D$ will represent the same transformation as $A$, just in the language of the eigenbasis.”

If $S$ consists of the eigenvectors of $A$, we are guaranteed to have a diagonal matrix with eigenvalues down the diagonal. → This is because we are then working in a coordinate system where the basis vectors get scaled by the transformation!

Then we can compute $A^n$ like so:

$$A^n=S(D^n)S^{-1}$$

Not every $A$ has enough eigenvectors to form proper eigenspace — the eigenspace needs to have the same dimension as the original space (rank $n$) – so this trick is not always possible, like for a shear.
Symmetric matrices are always diagonalizable.

Link to original

Eigen decomposition

For a square matrix $A\in {\mathbb{R}}^{n\times n}$, if $A$ has $n$ linearly independent eigenvectors, it can be decomposed as:

$$A=Q\Lambda Q^{-1}=\left[\begin{array}{cccc}|& |& & |\\ v_1& v_2& \cdots & v_n\\ |& |& & |\end{array}\right]\left[\begin{array}{cccc}{\lambda }_1& & & \\ & {\lambda }_2& & \\ & & \ddots & \\ & & & {\lambda }_n\end{array}\right]{\left[\begin{array}{cccc}|& |& & |\\ v_1& v_2& \cdots & v_n\\ |& |& & |\end{array}\right]}^{-1}$$

where $v_k$ are the linearly independent eigenvectors of $A$, and ${\lambda }_k$ are the corresponding eigenvalues. $Q$ is the matrix of eigenvectors, and $\Lambda$ is a diagonal matrix of eigenvalues.
Eigendecomposition is a type of matrix factorization

Geometric Interpretation

The eigendecomposition reveals how $A$ transforms space:
$Q^{-1}$ changes to the eigenvector basis (“rotate $I_n$ to $Q$”)
$\Lambda$ scales along each eigenvector direction by ${\lambda }_k$
$Q$ transforms back to the original basis (“rotate $Q$ back to $I_n$”)

Not all square matrices have an eigendecomposition!

A matrix must have $n$ linearly independent eigenvectors (complete eigenbasis; rank $n$) to be diagonalizable. Matrices that don’t satisfy this are called defective.

Spectral theorem

Every symmetric matrix is diagonalizeable.

---

Let $A\in {\mathbb{R}}^{n\times n}$ be a symmetric matrix. Then, all eigenvalues ${\lambda }_1,\dots ,{\lambda }_n$ of $A$ are real, and there exists an orthonormal basis of ${\mathbb{R}}^n$ consisting of eigenvectors of $A$ (aka eigenbasis).
Then:

$$A=A^T⟹A=Q\Lambda Q^T$$

$Q$ … orthogonal matrix ¹² ($Q^{-1}=Q^T$) whose columns are the eigenvectors of $A$
$\Lambda =\text{diag}({\lambda }_1,\dots ,{\lambda }_n)$
→ For symmetric matrices, this is a pure scaling in the orthogonal directions of the eigenvectors or principal axes - the natural directions along which $A$ acts purely by scaling. Each eigenvector direction $v_k$ is scaled by its eigenvalue ${\lambda }_k$ without any rotation or shearing.
→ Eigendecomposition and SVD coincide for symmetric matrices.

Link to original

I think some of the nuances here are incorrect.

Symmetric-Matrices-and-Eigendecomposition.pdf

https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix
https://en.wikipedia.org/wiki/Diagonalizable_matrix
https://en.wikipedia.org/wiki/Spectral_theorem