universal approximation theorem

Universal approximation theorem

A feedforward ANN with a single hidden layer containing a finite number of neurons can approximate any continuous function on compact subsets of ${\mathbb{R}}^n$ to arbitrary accuracy, given a suitable activation function.

The theorem says such networks can represent any continuous function, not that gradient descent will find the right weights, or that the required width is practical.