matrix

$A_{m\times n}$
$m$ … rows
$n$ … columns

$$\left[\begin{array}{ccc}{a}_{1,1}& a_{1,2}& a_{1,3}\\ a_{2,1}& a_{2,2}& a_{2,3}\\ a_{3,1}& a_{3,2}& a_{3,3}\end{array}\right]$$

($=A_{3\times 3}$)
The main diagonal goes from top left to bottom right.

Matrix operations

Addition and subtraction

Let $A$ and $B$ be two 2x2 matrices:

$$A=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\text{and}B=\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]$$

Their elementwise addition / subtraction, denoted by $C=A\pm B$, is given by:

$$C=\left[\begin{array}{cc}{a}_{11}\pm b_{11}& a_{12}\pm b_{12}\\ a_{21}\pm b_{21}& a_{22}\pm b_{22}\end{array}\right]$$

Scalar multiplication

Let $A$ be a 2x2 matrix and $\lambda$ be a scalar:

$$A=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]$$

Scalar multiplication, denoted by $B=\lambda A$, is given by:

$$B=\left[\begin{array}{cc}\lambda a_{11}& \lambda a_{12}\\ \lambda a_{21}& \lambda a_{22}\end{array}\right]$$

Matrix multiplication

Let $A$ and $B$ be two 2x2 matrices:

$$A=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\text{and}B=\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]$$

Matrix multiplication, denoted by $C=AB$, is given by:

$$C=\left[\begin{array}{cc}{a}_{11}{b}_{11}+a_{12}{b}_{21}& a_{11}{b}_{12}+a_{12}{b}_{22}\\ a_{21}{b}_{11}+a_{22}{b}_{21}& a_{21}{b}_{12}+a_{22}{b}_{22}\end{array}\right]$$

The entry $C_{i,j}$ comes from row $i$ of $A$ and column $j$ of $A$ (the dot product $A_i\cdot B_j$).
E.g.:

$$C_{1,2}=a_{11}{b}_{12}+a_{12}{b}_{22}=\sum _{k=1}^n{a}_{1,k}{b}_{k,2}$$

Visually: Matrix Multiplication.html

Intuitions

Viewing it from a col / row perspectve

Multiplying a matrix with a vector, results in a new vector, which is a linear combination of the columns of that matrix:

$$\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\\ 7& 8& 9\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{c}1x+2y+3z\\ 4x+5y+6z\\ 7x+8y+9z\end{array}\right]$$

We have $x\times \text{column 1}+y\times \text{column 2}+\dots$

Multiplying a row with a matrix gives us a new row, which is a linear combinations of the rows of that matrix:

$$\left[\begin{array}{ccc}x& y& z\end{array}\right]\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\\ 7& 8& 9\end{array}\right]=\left[\begin{array}{ccc}1x+4y+7z& 2x+5y+8z& 3x+6y+9z\end{array}\right]={\left[\begin{array}{c}1x+4y+7z\\ 2x+5y+8z\\ 3x+6y+9z\end{array}\right]}^T$$

We have $x\times \text{row 1}+y\times \text{row 2}+\dots$

Let’s multiply a row by a column:

$$\left[\begin{array}{c}2\\ 3\\ 4\end{array}\right]\left[\begin{array}{cc}1& 6\end{array}\right]=\left[\begin{array}{cc}2& 12\\ 3& 18\\ 4& 24\end{array}\right]$$

We can see that the resulting rows are multiples of the rows of $B$ and the resulitng colums are multiples of the columns of $A$.

If plotted, all the rows of $C$ would be on the same line, the one of the vector [1 6].
The row-space (all possible linear combinations of rows of that matrix) is a line. The same goes for the column-space. ¹

Viewing it as: AB = sum of ((cols of A) x (rows of B))

$$\left[\begin{array}{cc}2& 7\\ 3& 8\\ 4& 9\end{array}\right]\left[\begin{array}{cc}1& 6\\ 0& 0\end{array}\right]=\left[\begin{array}{c}2\\ 3\\ 4\end{array}\right]\left[\begin{array}{cc}1& 6\end{array}\right]+\left[\begin{array}{c}7\\ 8\\ 9\end{array}\right]\left[\begin{array}{cc}0& 0\end{array}\right]=\left[\begin{array}{cc}2& 12\\ 3& 18\\ 4& 24\end{array}\right]$$

Block matmul

We can also for example cut two 16x16 matrices both into 4x 4x4 matrices, multiply those individually and will still get the same result (example).

---

Which permutation matrix flips the rows of a matrix, such that $P\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]=\left[\begin{array}{cc}c& d\\ a& b\end{array}\right]$?
Well, we first want 0 of the first row, 1 of the second row. And then 1 of the first row and zero of the second row, so: $P=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]$. It is the opposite of the Identity matrix.

Which matrix flips the columns of the matrix, such that $P\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]=\left[\begin{array}{cc}b& a\\ d& c\end{array}\right]$?
Answer: None. If we multiply on the left of the matrix, we are always doing row operations. To switch the columns, then: $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\underset{P}{\underbrace{\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]}}=\left[\begin{array}{cc}b& a\\ d& c\end{array}\right]$.
To do col ops, the matrix multiplies on the right, to do row ops, the matrix multiplies on the left.

Rules

1. Dimensions: Ensure $A$ has dimensions $m\times n$ and $B$ has dimensions $n\times p$. The resulting matrix $C$ will have dimensions $m\times p$.
2. Order matters: In general, $AB\ne BA$. (Not commutative)
3. Associative property: $(AB)C=A(BC)$ for any matrices $A$, $B$, and $C$ with compatible dimensions.
4. Distributive property: $A(B+C)=AB+AC$ and $(A+B)C=AC+BC$ for matrices with compatible dimensions.

Division

Division is not defined because matrix multiplication is not commutative and there is not always an inverse to a matrix.
So if the division two matrices $A_{2\times 2}$ and $B_{2\times 2}$ is defined as:

$$C=A/B=A{B}^{-1}$$

If $B^{-1}$ is not defined, we have a problem.

Transpose

Given a matrix $A$, its transpose $A^T$ is obtained by flipping the matrix over its diagonal. This swaps the row and column indices of each element. If

The transpose operation has the following properties:

1. The transpose of a transpose is the original matrix: $(A^T{)}^T=A$.
2. The transpose of a sum is the sum of the transposes: $(A+B{)}^T=A^T+B^T$.
3. The transpose of a product is the product of the transposes in reverse order: $(AB{)}^T=B^T{A}^T$.
4. The transpose of a scalar multiple is the scalar multiple of the transpose: $(\lambda A{)}^T=\lambda (A^T)$.

Note: The diagonal of a matrix remains the same in the transpose.

determinant

Types of matrices

Row matrix

A matrix with only one row and any number of columns.

$$R=\left[\begin{array}{ccc}1& 2& 3\end{array}\right]$$

Column matrix

A matrix with only one column and any number of rows. Aka vector, aka column.

$$C=\left[\begin{array}{c}4\\ 5\\ 6\end{array}\right]$$

Zero matrix

A matrix in which all the elements are zero.

$$Z=\left[\begin{array}{cc}0& 0\\ 0& 0\end{array}\right]$$

Square matrix

A matrix with an equal number of rows and columns.

$$S=\left[\begin{array}{cc}7& 8\\ 9& 10\end{array}\right]$$

diagonal matrix

A matrix with nonzero elements only on the main diagonal.

$$D=\left[\begin{array}{cc}a& 0\\ 0& b\end{array}\right]$$

Identity matrix

A square matrix with 1s on the main diagonal and 0s elsewhere. Any matrix multiplied by its identity matrix results in the same matrix: $$
A = I_mA = AI_n
$$$\forall \space A \in \mathbb{R}^{m \times n}I_m$/$I_n$aretheleft/rightidentitymatricesofthesizeof$A$‘srows/colsrespectively.\ast Ofcourse\ast inGerman,thereisadifferentnotation,$I \equiv E$ (for “Einheitsmatrix”).

$$I=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$$

Triangular matrix

An upper/lower triangular matrix is a square matrix, where all the elements below/above the main diagonal are zero, e.g.:

$$U=\left[\begin{array}{cc}1& 1\\ 0& 1\end{array}\right]L=\left[\begin{array}{cc}1& 0\\ 1& 1\end{array}\right]$$

References

Gilbert Strang OCW lectures

Visual Kernel YT Channel with visualizations of different matrices, spectral decomposition, SVD, …

Footnotes

1. See Linear dependance for an example. ↩