Given a training set \(\{({\textcolor{input}{\boldsymbol{x}}}_n, \textcolor{output}{y_n})\}_{n=1}^N\), we want to predict the label \(\textcolor{output}{y}_\star\) of a new input \({\textcolor{input}{\boldsymbol{x}}}_\star\).
Probabilistic classifiers: output a probability distribution over the labels \(P(\textcolor{output}{y}_\star = k \mid {\textcolor{input}{\boldsymbol{x}}}_\star)\).
For binary classification, \(P(\textcolor{output}{y}_\star = 1 \mid {\textcolor{input}{\boldsymbol{x}}}_\star)\) and \(P(\textcolor{output}{y}_\star = 0 \mid {\textcolor{input}{\boldsymbol{x}}}_\star)\).
Non-probabilistic classifiers: produce a hard assignment \(\textcolor{output}{y}_\star = k\).
Examples: Probabilistic classifier
Logistic regression
Naive Bayes
Examples: Non-probabilistic classifier
Support Vector Machines
Decision Trees
K-Nearest Neighbors
Probabilistic classifiers
Probabilistic classifiers are more informative than non-probabilistic classifiers.
\(P(\textcolor{output}{y}_\star = 1 \mid {\textcolor{input}{\boldsymbol{x}}}_\star) = 0.7\) is more informative than \(\textcolor{output}{y}_\star = 1\).
\(P(\textcolor{output}{y}_\star = 1 \mid {\textcolor{input}{\boldsymbol{x}}}_\star)\) gives us a measure of confidence in the prediction.
Particularly useful in applications where the cost of misclassification is high.
Logistic Regression
Logistic regression is a probabilistic classifier that models the probability of the output \(\textcolor{output}{y}\) given the input \({\textcolor{input}{\boldsymbol{x}}}\).
We model \(P(\textcolor{output}{y} = 1 \mid {\textcolor{input}{\boldsymbol{x}}})\) through some function \(f({\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}})\).
In linear regression, \(f({\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}}) = {\textcolor{params}{\boldsymbol{w}}}^\top{\textcolor{input}{\boldsymbol{x}}}\). Can we use this for classification?
No: the output of linear regression is unbounded and it can’t be interpreted as a probability.
But, we can use a function \(h(\cdot)\) to map the output of linear regression to the interval \([0, 1]\).
Logistic function
For logistic regression, we use the sigmoid function
We place a prior distribution over the parameters \({\textcolor{params}{\boldsymbol{w}}}\) and we define a likelihood to obtain the posterior distribution.
For linear regression, we could find \({\textcolor{params}{\boldsymbol{w}}}_{\text{MAP}}\) in closed form.
For logistic regression, we can’t do this in closed form.
We can use numerical optimization methods to find \({\textcolor{params}{\boldsymbol{w}}}_{\text{MAP}}\):
Gradient ascent (or descent)
Newton-Raphson’s method
Gradient descent
We can use gradient descent to find the mode of the posterior distribution.
We need to compute the gradient \(\grad_{{\textcolor{params}{\boldsymbol{w}}}} {\mathcal{U}}({\textcolor{params}{\boldsymbol{w}}})\) and iterate until convergence.
Approximate the posterior distribution with a Gaussian distribution \(q({\textcolor{params}{\boldsymbol{w}}}) = {\mathcal{N}}({\textcolor{params}{\boldsymbol{w}}}\mid{\boldsymbol{\mu}}, {\boldsymbol{\Sigma}})\).
Find the mode of the posterior distribution \({\textcolor{params}{\boldsymbol{w}}}_{\text{MAP}}\) (we have this already).
Compute the Hessian of the log posterior at \({\textcolor{params}{\boldsymbol{w}}}_{\text{MAP}}\).
The covariance of the Gaussian approximation is given by the inverse of the Hessian.
Laplace Approximation
Making predictions
We can make predictions by integrating over the posterior distribution. \[
p(\textcolor{output}{y}_\star \mid {\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}}) = \int p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}}_\star)q({\textcolor{params}{\boldsymbol{w}}})\dd{\textcolor{params}{\boldsymbol{w}}}
\]
Even though we have a Gaussian approximation of the posterior, this integral is intractable.
Two common approximations:
Monte Carlo integration: sample from the posterior distribution and average the predictions. \[
p(\textcolor{output}{y}_\star \mid {\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}}) \approx \frac{1}{S}\sum_{s=1}^S p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}^{(s)}, {\textcolor{input}{\boldsymbol{x}}}_\star) \quad \text{where } {\textcolor{params}{\boldsymbol{w}}}^{(s)} \sim q({\textcolor{params}{\boldsymbol{w}}})
\]
Probit approximation: use deterministic approximation of the sigmoid function.
Making predictions with probit approximation
Logistic function \(\sigma(t)\) is approximated by the cumulative distribution function of a Gaussian \(\Phi(\lambda t)\), where \(\lambda = \sqrt{\pi/8}\).
Making predictions with probit approximation
We can compute the distribution of \(\textcolor{latent}{f}_\star = f({\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}}_\star)\) where \(f({\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}}) = {\textcolor{params}{\boldsymbol{w}}}^\top{\textcolor{input}{\boldsymbol{x}}}\) before applying the sigmoid function.
The distribution of \(\textcolor{latent}{f}_\star\) is a Gaussian distribution. \[
q(\textcolor{latent}{f}_\star \mid {\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}}) = \int p(\textcolor{latent}{f}_\star \mid {\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}}_\star){\mathcal{N}}({\textcolor{params}{\boldsymbol{w}}}\mid{\boldsymbol{\mu}}, {\boldsymbol{\Sigma}})\dd{\textcolor{params}{\boldsymbol{w}}}= {\mathcal{N}}(\textcolor{latent}{f}_\star\mid{\boldsymbol{\mu}}^\top{\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{input}{\boldsymbol{x}}}_\star^\top{\boldsymbol{\Sigma}}{\textcolor{input}{\boldsymbol{x}}}_\star)
\]
Define a family of distributions \(q({\textcolor{params}{\boldsymbol{w}}}; {\textcolor{vparams}{\boldsymbol{\nu}}})\), where \({\textcolor{vparams}{\boldsymbol{\nu}}}\) are the variational parameters.
Minimize the KL divergence between the true posterior and the variational distribution
Even though we can’t compute the posterior distribution, we can sample from it.
Markov Chain Monte Carlo (MCMC) methods are a class of algorithms that allow us to sample from complex distributions.
MCMC methods are based on the idea of proposing new samples and accepting them with a certain probability:
Random walk: propose a new sample by adding Gaussian noise to the current sample.
Hamiltonian Monte Carlo: propose a new sample by simulating the dynamics of a physical system.
Sampling from the posterior with MCMC
Has it converged?
Run multiple chains with different initializations.
Check for convergence using trace plots and density estimates.
Predictions with MCMC
Which method to use?
No free lunch theorem: no method is the best for all problems.
MAP estimation: fast, but doesn’t provide uncertainty estimates.
Laplace approximation: needs to compute the MAP estimate and the Hessian inverse (can be very expensive for large models).
Variational inference: flexible, scalable to large models and datasets, but rough approximation (unless you use more complex distributions).
MCMC: most accurate, but computationally expensive
Random walk Metropolis: simple, but slow convergence.
Hamiltonian Monte Carlo: faster convergence, but requires gradient computation.
Uncertainty estimates
Uncertainty on class predictions
One advantage of Bayesian methods is that they provide uncertainty estimates.
We have seen how to average all the decision boundaries for \(P(\textcolor{output}{y}_\star = 1 \mid {\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}})\), by weighting them by the posterior distribution.
But we can also compute the variance of the predictions, which gives us an idea of the uncertainty on the class probabilities, e.g. \(P(\textcolor{output}{y}_\star = 1 \mid {\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}}) = 0.7 \pm 0.1\).
Uncertainty on class predictions (variance)
We can compute the variance of the class probabilities
Performance evaluation
Evaluating the performance of a classifier
Accuracy: proportion of correctly classified examples.
Error rate (or 0/1 loss): proportion of misclassified examples.
Consider a set of predictions \(\widehat{\textcolor{output}{y}}_i\) and true labels \(\textcolor{output}{y}_i\) for \(i=1,\ldots,N\).
\(\text{Precision} = 0\) (no fraudulent transactions are detected).
\(\text{Recall} = 1\) (all non-fraudulent transactions are correctly classified).
\(\text{F1 score} = 0\).
Predictive test log-likelihood
So far, we have evaluated the performance of the classifier based on the class predictions not the probabilities.
Predictive log-likelihood: test the ability of the model to correctly predict the class probabilities.
Recall the predictive distribution \(p(\textcolor{output}{y}_\star \mid {\textcolor{input}{\boldsymbol{x}}}_\star, {\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}})\) after marginalizing over the parameters with the posterior distribution (or any approximation).
where \({\textcolor{params}{\boldsymbol{w}}}^{(s)} \sim p({\textcolor{params}{\boldsymbol{w}}}\mid{\textcolor{output}{\boldsymbol{y}}}, {\textcolor{input}{\boldsymbol{X}}})\).
Problem: we cannot swap log and sum, and we can have numerical issues when computing the the likelihood for each sample \(p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}^{(s)}, {\textcolor{input}{\boldsymbol{x}}}_\star)\).
Log-sum-exp trick
We can use the log-sum-exp trick to avoid numerical issues. \[
\begin{aligned}
\log\sum_{s=1}^S p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}^{(s)}, {\textcolor{input}{\boldsymbol{x}}}_\star) &= \log\sum_{s=1}^S \exp\left(\log p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}^{(s)}, {\textcolor{input}{\boldsymbol{x}}}_\star)\right)\\
&= \text{logsumexp}\left(\log p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}^{(1)}, {\textcolor{input}{\boldsymbol{x}}}_\star), \ldots, \log p(\textcolor{output}{y}_\star \mid {\textcolor{params}{\boldsymbol{w}}}^{(S)}, {\textcolor{input}{\boldsymbol{x}}}_\star)\right)
\end{aligned}
\]
Many libraries (Numpy, Pytorch, JAX, etc.) provide a numerical stable implementation of the log-sum-exp function
The trick is based on the following identity: \[
\log(\exp(a) + \exp(b)) = a + \log(1 + \exp(b - a))
\]
with \(a \ge b\).
Calibration
Calibration: the predicted probabilities should match the true probabilities.
When we predict a class probability of 0.9 for a class, we expect that 90% of the examples with this predicted probability belong to this class.
Tools to evaluate calibration:
Expected calibration error (ECE)
Reliability diagram
Accuracy and confidence
To compute calibration metrics, we need to group the examples based on the predicted probabilities.
Define:
\(\textcolor{output}{y}_i = \arg\max_k p(\textcolor{output}{y}_i = k \mid {\textcolor{input}{\boldsymbol{x}}}_i)\)
\(\widehat{p}_i = \max_k p(\textcolor{output}{y}_i = k \mid {\textcolor{input}{\boldsymbol{x}}}_i)\)
Now:
Divide the range of predicted probabilities into \(B\) buckets or bins.
Let \(\mathcal B_b\) be the set of examples with predicted probabilities in \(\left(\frac{b-1}{B}, \frac{b}{B}\right]\).
Temperature scaling: scale the logits before applying the sigmoid function. \[
p(\textcolor{output}{y}_i \mid {\textcolor{params}{\boldsymbol{w}}}, {\textcolor{input}{\boldsymbol{x}}}_i) = \text{Bern}(\sigma({\textcolor{params}{\boldsymbol{w}}}^T{\textcolor{input}{\boldsymbol{x}}}_i/T))
\]
Improving calibration: a recipe for temperature scaling
Train the model.
Compute the ECE on the validation set.
Choose the temperature that minimizes the ECE on the validation set.
Re-calibrate the model using the chosen temperature.
Note: temperature scaling is not affecting model performance (accuracy, F1 score, etc.), but only the calibration.
