generalization error

The generalizatino error or risk is the exptected loss on future data for a given model $f(\cdot ;\theta )$ with parameters $\theta$, defined as:

$$\begin{array}{rl}R(f(\cdot ;\theta ))& =\mathbb{E}[\mathcal{L}(f(x;\theta ),y)]\\ & ={\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}\mathcal{L}(f(x;\theta ),y)p(x,y)dydx\end{array}$$

where $\mathcal{L}(\hat{y},y)$ is the loss function, and $p(x,y)$ is the true (unknown) joint distribution of inputs $x$ and outputs $y$.

We estimate the generalization error using the empirical risk (or training error) computed on a finite training dataset $\{(x_i,y_i){\}}_{i=1}^n$:

$$\hat{R}(f(\cdot ;\theta ))=\frac{1}{n}\sum _{i=1}^n\mathcal{L}(f(x_i;\theta ),y_i)$$

This works under the assumption that the training data is drawn iid from the same distribution as future data, and by the law of large numbers, the empirical risk converges to the true risk as $n\to \infty$.

If we train plain empirical risk minimization (ERM), we run the risk of overfitting, where the model fits the training data well (low empirical risk) but performs poorly on unseen data (high generalization error) .
→ Split data into training and validation/test sets to estimate generalization error, e.g. using cross-validation.