Logistic regression / Sigmoid

• Logistic regression predicts a probability that an input belongs to class 1 (positive class).
• It uses the sigmoid (logistic) function to squash (transform) any real number into a value between 0 and 1.
• Finally we convert that probability to a class using a threshold (usually 0.5).
• Although called “regression,” it is a classification method.

Equations:

$$ \begin{align*} \sigma(z) &= \frac{1}{1 + e^{-z}} \\[6pt] \hat{p}(x) &= \sigma(w_0 + w_1 x_1 + \dots + w_p x_p) \end{align*} $$

Decision rule:

$$ \begin{align*}\text{Predict } \hat{y} &=\begin{cases}1 & \text{if } \hat{p}(x) \ge 0.5 \\[4pt]0 & \text{if } \hat{p}(x) < 0.5\end{cases}\end{align*} $$

Logistic function

$$ \begin{align*} y &= f(n) &= \frac{1}{1 + e^{-z}} \end{align*} $$

• Here,

  $e = \text{base of the natural logarithm} \\ e = 2.71828$

• The logistic function is also called as Sigmoid function in neural networks.

Essentials:

• $\sigma(z)$ always outputs a value in $(0,1)$.
• $\hat{p}(x)$ = predicted probability for class 1.
• Threshold 0.5 is common but we can change it (e.g., to trade precision vs recall).