Logistic Regression is one of the simplest supervised learning algorithms for binary classification. Given an input feature vector, it computes a linear score, maps that score into a probability using the sigmoid function, and then classifies the sample based on a decision threshold.
Despite the name, Logistic Regression is a classification model, not a regression model. It is called regression because it learns a linear scoring function, but the sigmoid function turns that score into a probability ranging from 0 to 1, enabling classification decisions. It remains one of the most popular and reliable classification techniques in production because of its mathematical simplicity and efficiency.
Core Concepts
- Sigmoid Function: Maps any real-valued number into a probability between 0 and 1:
- Binary Cross-Entropy (BCE) Loss: The objective function used to penalize wrong classifications heavily based on confidence. It assigns a steep penalty when the model makes a highly confident but incorrect prediction:
- Decision Boundary: The structural hyperplane where the probability output equals a chosen threshold, typically defaulted to 0.5 (i.e., ).
Mathematical Derivation of the Gradient
We will derive the gradient of Binary Cross-Entropy with respect to the linear score. The mathematical elegance of Logistic Regression comes from how the derivative of the Sigmoid activation perfectly simplifies the derivative of the Cross-Entropy loss.
1. Derivative of the Sigmoid Function
Let
Using the quotient rule:
2. Derivative of the BCE Loss with respect to
For a single sample, . Differentiating with respect to :
3. Applying the Chain Rule to find
Combine the two derivatives to find the gradient with respect to the raw linear model output (
z):As you can see, the gradient with respect to the logit (
z) reduces to . While the Sigmoid activation introduces an additional derivative term, this term is exactly canceled by the derivative of the BCE loss through the chain rule. Consequently, the final gradient depends only on the prediction error (). This elegant result is one of the main reasons why the Sigmoid activation is commonly paired with BCE loss, as it leads to a simple, efficient, and numerically stable optimization process.Implementation
Logistic Regression starts with the same linear score as Linear Regression:Then it applies the sigmoid function to map the score into a probability between 0 and 1.During training, we minimize Binary Cross-Entropy. The key simplification is that, for sigmoid activation paired with BCE loss, the gradient with respect to the logit becomes:So each gradient descent step computes probabilities, subtracts the labels to get the error vector, projects that error back through the feature matrix with , and updates the weights and bias in the opposite direction of the gradient.
Note that the gradient formulas for logistic regression with BCE loss have the same form as those for linear regression with mean squared error (MSE) loss. This occurs because the derivative of the Sigmoid activation is canceled by the derivative of the BCE loss during backpropagation. Therefore, the update equations look exactly the same; only the prediction（
y_pred）differs.Algorithmic Complexity
Let
n be the number of sample observations, d be the number of features, and T be the total training iterations.- Time Complexity:
- In each training iteration, the forward pass dot-product scales at . The gradient computation multiplies the transposed matrix by our error vector, running at .
- Space Complexity:
- The memory footprint is highly optimized. Excluding the input matrices, the model stores
dweights and one bias, and each training step materializes prediction and error vectors of lengthn.
FAQs
Q1: Why doesn't Logistic Regression use Mean Squared Error (MSE) as its loss function?
With a sigmoid output, MSE does not produce the same clean convex objective as BCE for Logistic Regression. It also introduces an extra factor into the gradient, which can make learning very slow when the sigmoid saturates. This creates an optimization landscape with many local minima and flat regions, meaning gradient descent can easily get trapped in a suboptimal solution.
Additionally, using MSE causes severe gradient disappearance (or gradient saturation). The derivative of MSE contains a term for the derivative of the Sigmoid function (). If the model makes a highly confident but incorrect prediction, the Sigmoid function saturates at its flat outer edges, where the gradient drops to nearly zero. This prevents the model from updating its weights when it is wrong. Binary Cross-Entropy eliminates this saturation effect, ensuring that larger errors generate larger gradients to drive faster learning.
Q2: How do you adapt Logistic Regression to handle Multi-Class classification tasks?
There are two main strategies for scaling Logistic Regression to multi-class problems:
- One-vs-Rest (OvR / One-vs-All): For a dataset with
Kunique classes, we trainKindependent binary logistic regression models. Each model learns to distinguish one specific class from all other classes combined. During inference, we run the sample through allKclassifiers and pick the class that outputs the highest confidence score.
- Softmax Regression (Multinomial Logistic Regression): We modify the architecture directly by expanding the parameter space to include a vector of weights for each class. We then replace the single binary Sigmoid function with a multi-class Softmax function, which transforms the vector of raw linear scores into a valid probability distribution over all
Kclasses simultaneously.
Q3: If a Logistic Regression model overfits, what strategies can you use to fix it?
To mitigate overfitting in Logistic Regression, we can apply several techniques:
- Regularization: We can add a penalty term to our objective function. L1 Regularization (Lasso) adds a penalty proportional to the absolute values of the weights (), which drives less important feature weights to exactly zero, performing implicit feature selection. L2 Regularization (Ridge) adds a penalty proportional to the squared values of the weights (), which shrinks the weights toward zero to prevent individual features from dominating.
- Feature Selection: We can manually or programmatically drop highly correlated or noisy features using tools like variance thresholds or correlation matrices to reduce multi-collinearity.
- More data and better preprocessing: More representative data reduces variance. Feature scaling also improves optimization behavior for Gradient Descent, although it does not directly prevent statistical overfitting by itself.
- Author:Fan Luo
- URL:https://fanluo.me/article/logistic-regression-from-scratch
- Copyright:All articles in this blog adopt BY-NC-SA agreement. Please indicate the source!
上一篇
K-Means Clustering
下一篇
Linear Regression from Scratch