eigendecomposition

Diagonalization: Performing a transformation from the perspective of the eigenbasis.

We can exploit the efficiency of diagonal matrices for taking large powers if we convert a transformation matrix $A$ into a diagonal one by multiplying it with the change-of-basis matrix $S$ and its inverse:

$$D=S^{-1}AS$$

“First translate from the language of the eigenbasis to our basis, then apply $A$, then translate back. The resulting matrix $D$ will represent the same transformation as $A$, just in the language of the eigenbasis.”

If $S$ consists of the eigenvectors of $A$, we are guaranteed to have a diagonal matrix with eigenvalues down the diagonal. → This is because we are then working in a coordinate system where the basis vectors get scaled by the transformation!

Then we can compute $A^n$ like so:

$$A^n=S(D^n)S^{-1}$$

Not every $A$ has enough eigenvectors to form proper eigenspace — the eigenspace needs to have the same dimension as the original space (rank $n$) – so this trick is not always possible, like for a shear.
Symmetric matrices are always diagonalizable.

Link to original

Eigen decomposition

For a square matrix $A\in {\mathbb{R}}^{n\times n}$, if $A$ has $n$ linearly independent eigenvectors, it can be ecomposed decomposed as:

$$A=Q\Lambda Q^{-1}=\left[\begin{array}{cccc}|& |& & |\\ v_1& v_2& \cdots & v_n\\ |& |& & |\end{array}\right]\left[\begin{array}{cccc}{\lambda }_1& & & \\ & {\lambda }_2& & \\ & & \ddots & \\ & & & {\lambda }_n\end{array}\right]{\left[\begin{array}{cccc}|& |& & |\\ v_1& v_2& \cdots & v_n\\ |& |& & |\end{array}\right]}^{-1}$$

where $v_k$ are the linearly independent eigenvectors of $A$, and ${\lambda }_k$ are the corresponding eigenvalues. $Q$ is the matrix of eigenvectors, and $\Lambda$ is a diagonal matrix of eigenvalues.

Geometric Interpretation

The eigendecomposition reveals how $A$ transforms space:
$Q^{-1}$ changes to the eigenvector basis (“rotate $I_n$ to $Q$”)
$\Lambda$ scales along each eigenvector direction by ${\lambda }_k$
$Q$ transforms back to the original basis (“rotate $Q$ back to $I_n$”)

I think some of the nuances here are incorrect.

https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix
https://en.wikipedia.org/wiki/Diagonalizable_matrix
https://en.wikipedia.org/wiki/Spectral_theorem

Spectral Theorem

Every symmetric matrix is orthogonally diagonalizeable:

$$A=A^T⟹A=Q\Lambda Q^T$$

where $Q$ is an orthogonal matrix ¹² ($Q^{-1}=Q^T$).
→ Eigendecomposition and SVD coincide for symmetric matrices.
→ For symmetric matrices, this is a pure scaling in the orthogonal directions of the eigenvectors or principal axes - the natural directions along which $A$ acts purely by scaling. Each eigenvector direction $v_k$ is scaled by its eigenvalue ${\lambda }_k$ without any rotation or shearing.

Not all square matrices have an eigendecomposition!

A matrix must have $n$ linearly independent eigenvectors (complete eigenbasis; rank $n$) to be diagonalizable. Matrices that don’t satisfy this are called defective.

Derivation from the property of eigenvalues

$$\begin{array}{rl}Av& =\lambda v\\ AQ& =Q\Lambda \\ A& =Q\Lambda Q^{-1}\end{array}$$

Footnotes

1. It doesn’t matter if $Q$ is normalized, $Q^{-1}$ cancels the magnitude. ↩

2. For symmetric matrices, the eigenvectors are orthogonal, and if normalized, $Q$ becomes orthogonal ($Q^{-1}=Q^T$). However, for general diagonalizable matrices, eigenvectors need only be linearly independent. ↩