logistic regression

Logistic Regression

A supervised learning model that applies the sigmoid function to a linear model, producing a probability estimate:

$$\hat{y}=\sigma ({\mathbf{w}}^T\mathbf{x}+b)=\frac{1}{1+e^{-({\mathbf{w}}^T\mathbf{x}+b)}}$$

where $\mathbf{w}\in {\mathbb{R}}^d$ are weights, $b$ is a bias, and $\hat{y}\in (0,1)$ is interpreted as $P(y=1|\mathbf{x})$.

Logistic regression is not a classification algorithm on its own - the output is a probability, a real value in $[0,1]$. It becomes a classifier when combined with a decision rule, e.g., predict class 1 if $\hat{y}>0.5$.

Why not just use linear regression for classification?

Linear regression outputs unbounded values - a point far from the decision boundary might get predicted as $y=47$, which is meaningless for class probabilities. The sigmoid squashes everything into $(0,1)$, giving proper probabilities that sum correctly.

The key difference is the nonlinearity: logistic regression is essentially the simplest possible neural network - a linear model followed by one nonlinear activation function. Stack more layers with nonlinearities and you get a multi-layer perceptron. Sigmoid is the classic choice, but other functions work too.

Optimization

Unlike linear regression, logistic regression has no closed-form solution. We can use gradient descent:

$${\mathbf{w}}_{t+1}={\mathbf{w}}_t-\eta \frac{\partial L}{\partial \mathbf{w}}$$

With cross-entropy loss, the problem is convex (single global minimum). MSE would make it non-convex: the sigmoid’s flat tails create near-zero gradients when predictions are confidently wrong, trapping gradient descent. Cross-entropy penalizes confident wrong predictions heavily, keeping gradients informative:

$$L=-\frac{1}{n}\sum _{i=1}^n\left[y_i\log ({\hat{y}}_i)+(1-y_i)\log (1-{\hat{y}}_i)\right]$$

Multi-class extension

For $K>2$ classes, replace sigmoid with softmax:

$$P(y=k|\mathbf{x})=\frac{e^{{\mathbf{w}}_k^T\mathbf{x}}}{{\sum }_{j=1}^K{e}^{{\mathbf{w}}_j^T\mathbf{x}}}$$

Each class gets its own weight vector. Softmax ensures probabilities sum to 1 across all classes.