maximum likelihood estimation

Maximum Likelihood Estimates for a Discrete Distribution

Given a set of observed values $x=(x_1,\dots x_N)$ of a discrete random variable $x$ with $\mathrm{Val}(x)=\{1,\dots ,k\}$, the maximum likelihood estimate for the parameters ${\hat{\theta }}_i=\hat{P}(x^i)$ of the probability distribution $P(X)=\{P(x^1),\dots ,P(X^k)\}$ is

$${\hat{\theta }}_i=\hat{P}(x^i)=\frac{1}{N}\sum _{j=1}^{N}1\{x_j=x^i\}$$

In other words: the MLE for the probability of an outcome is just the fraction of times it was observed in the data.
Relative frequences are optimal probability estimators, in the sense that they have maximum likelihood.

Issue with naively maximizing the likelihood: overfitting. Assigning probability 1 to the observed data, 0 to the unobserved data gives the maximum likelihood, at the cost of generalization.