(Work in progress I will gradually add more content when having more time:D Please stay tuned :D)

Machine Learning Paradigms

Supervised learning

Supervised learning is a type of machine learning where the model is trained on a labeled dataset, meaning that the input data is paired with the correct output. The model learns to make predictions based on this labeled data.

Mathematically, given a training set of \(N\) samples:

\[\mathcal{D} = \{(x_1, y_1), (x_2, y_2), \ldots, (x_N, y_N)\}\]

where \(x_i \in \mathbb{R}^d\) is the input feature vector with \(d\) dimensions. The corresponding label \(y_i\) can be either a continuous value \(y_i \in \mathbb{R}\) for regression problems, or a categorical value \(y_i \in \{0, 1, \ldots, K-1\}\) for classification problems.

the goal of supervised learning is to learn a function \(f_{\theta}\) parameterized by \(\theta\) that maps input features \(x\) to output labels \(y\), i.e., \(\hat{y} \triangleq f_{\theta}(x) \approx y\).

It can be done by minimizing the loss function - which measures the discrepancy between the predicted output \(\hat{y}\) and the true output \(y\) over the training set, i.e., \(\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} \mathcal{l}(y_i, f_{\theta}(x_i))\).

Common loss functions:

  • Mean squared error (MSE) for regression problems:
\[\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} (y_i - f_{\theta}(x_i))^2\]
  • Cross-entropy loss for classification problems:
\[\mathcal{L}(\theta) = - \frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} y_{ik} \log \hat{y}_{ik}\]

where \(y_{ik}\) is the one-hot encoded label for the \(i\)-th sample and the \(k\)-th class, and \(\hat{y}_{ik}\) is the predicted probability of the \(i\)-th sample for the \(k\)-th class.

More advanced topics which are discussed later in this post:

  • Advanced loss functions
  • Regularization
  • Bias-variance tradeoff
  • Out-of-distribution generalization
  • Adversarial generalization/training
  • Sharpness-aware minimization

Unsupervised learning

Unsupervised learning is a machine learning paradigm where a model learns patterns, structures, or representations from unlabeled data without explicit supervision.

Unlike supervised learning, there is no unified mathematical formulation for unsupervised learning, instead, it is a collection of different techniques/problems that can be grouped into following categories:

Dimensionality reduction/representation learning: Finding a low-dimensional representation of the data that captures the most important information, e.g., to learn a mapping \(f: \mathbb{R}^d \rightarrow \mathbb{R}^k\) where \(k \ll d\). Common methods include Principal Component Analysis (PCA), t-SNE, and Autoencoders.

Clustering: Finding groups of similar features, e.g., assigning cluster \(c_i\) to each sample \(x_i\), where \(c_i \in \{1, 2, \ldots, K\}\). Common methods include K-means, Gaussian Mixture Models (GMM), and Hierarchical Clustering.

Density estimation: Estimating the probability distribution \(p(x)\) of the data. Common methods include Parametric models (e.g., Gaussian Mixture Models (GMM), Mixture of Experts (MoE)), Non-parametric models (e.g., Kernel Density Estimation (KDE), Histogram), and Variational Autoencoders (VAEs).

Parametric methods are the ones that assume the data follows a known distribution (e.g., Gaussian, MoG, etc.) and characterize by a fixed number of parameters that do not grow with the amount of data.

Non-parametric methods are the ones that do not make any assumptions about the distribution of the data. The number of parameters can grow with more data. It provides greater flexibility but requires more data/computational expense.

Gaussian Mixture Model (GMM) models the data as a weighted sum of \(K\) Gaussian distributions:

\[p(x) = \sum_{k=1}^{K} \alpha_k \mathcal{N}(x; \mu_k, \Sigma_k)\]

where \(\alpha_k\) is the mixing coefficient, \(\mu_k\) is the mean, and \(\Sigma_k\) is the covariance matrix of the \(k\)-th Gaussian distribution. It can be learned using Expectation-Maximization (EM) algorithm, including the E-step and M-step:

E-step: compute the probability that each sample \(x_i\) belongs to each cluster \(k\):

\[p(c_i = k \mid x_i) = \frac{\alpha_k \mathcal{N}(x_i; \mu_k, \Sigma_k)}{\sum_{j=1}^{K} \alpha_j \mathcal{N}(x_i; \mu_j, \Sigma_j)}\]

M-step: update the parameters of the Gaussian distributions to maximize the likelihood of the data:

\[\alpha_k = \frac{1}{N} \sum_{i=1}^{N} p(c_i = k \mid x_i)\] \[\mu_k = \frac{\sum_{i=1}^{N} p(c_i = k \mid x_i) x_i}{\sum_{i=1}^{N} p(c_i = k \mid x_i)}\] \[\Sigma_k = \frac{\sum_{i=1}^{N} p(c_i = k \mid x_i) (x_i - \mu_k)(x_i - \mu_k)^T}{\sum_{i=1}^{N} p(c_i = k \mid x_i)}\]

And repeat the E-step and M-step until convergence.

Kernel Density Estimation (KDE) uses a kernel function to estimate the probability density function of the data.

\[p(x) = \frac{1}{Nh^d} \sum_{i=1}^{N} K \left( \frac{x - x_i}{h} \right)\]

where \(K\) is the kernel function, \(h\) is the bandwidth to control the smoothness of the estimated density, and \(d\) is the input dimension. For one-dimensional data, this reduces to \(\frac{1}{Nh}\). Common kernel functions include Gaussian, Epanechnikov, and Uniform kernels.

Self-supervised learning

Self-supervised learning is technically supervised learning in disguise: we create the labels from the data itself. This is why it is such a powerful idea. You do not need humans to annotate every image/text/audio clip, but you still train with a supervised-style objective.

Examples:

  • Predict the next token in language modeling.
  • Mask some image patches and reconstruct them, as in Masked Autoencoders.
  • Create two augmented views of the same image and pull their representations together, as in SimCLR.
  • Predict missing words, masked regions, future frames, or whether two chunks come from the same sample.

The general recipe is:

\[x \rightarrow (x_a, x_b) \rightarrow z_a = f_{\theta}(x_a), \quad z_b = f_{\theta}(x_b)\]

Then train the model so that related views are close and unrelated views are far away. A common contrastive loss is InfoNCE:

\[\mathcal{L}_i = - \log \frac{\exp(\mathrm{sim}(z_i, z_i^+) / \tau)} {\sum_{j=1}^{B} \exp(\mathrm{sim}(z_i, z_j) / \tau)}\]

where \(z_i^+\) is the positive example for \(z_i\), \(B\) is the batch size, \(\mathrm{sim}\) is usually cosine similarity, and \(\tau\) is the temperature.

💡 Intuition: self-supervised learning is about designing a “pretext task” that forces the model to learn useful structure. If the pretext task is too easy, the representation is weak. If it is too unrelated to the downstream task, the model learns the wrong thing very confidently, which is always a little rude :D

Clustering in representation space

Clustering can also be used as a self-supervised signal. Instead of assigning ground-truth labels, we assign pseudo-labels from cluster IDs:

\[c_i = \underset{k}{\mathrm{argmin}} \ \| f_{\theta}(x_i) - \mu_k \|^2\]

Then the model is trained to predict those cluster assignments. This is a useful trick when we believe the data has semantic groups, but we do not have labels yet.

Important caveat: clusters are not automatically “classes”. K-means might group images by background color, camera style, or lighting instead of the actual object. This is one of the classic places where ML looks smart and then quietly optimizes the wrong shortcut.

Semi-supervised learning

Semi-supervised learning sits between supervised and unsupervised learning. We have a small labeled dataset and a usually much larger unlabeled dataset:

\[\mathcal{D}_L = \{(x_i, y_i)\}_{i=1}^{N_L}, \quad \mathcal{D}_U = \{x_i\}_{i=1}^{N_U}, \quad N_U \gg N_L\]

The goal is to use unlabeled data to improve the model beyond what the labeled data alone can do.

Common approaches:

  • Self-training / pseudo-labeling: train on labeled data, predict labels for unlabeled data, then add high-confidence predictions back into training.
  • Consistency regularization: different augmentations of the same input should produce similar predictions.
  • Graph-based label propagation: build a similarity graph and propagate labels to nearby unlabeled samples.
  • Entropy minimization: encourage confident predictions on unlabeled samples.

The simplest pseudo-label objective:

\[\mathcal{L}(\theta) = \frac{1}{N_L} \sum_{(x,y) \in \mathcal{D}_L} \ell(f_{\theta}(x), y) + \lambda \frac{1}{N_U} \sum_{x \in \mathcal{D}_U} \ell(f_{\theta}(x), \hat{y})\]

where \(\hat{y} = \underset{k}{\mathrm{argmax}} \ f_{\theta}(x)_k\) is the pseudo-label.

⚠️ Important thing that people forget: pseudo-labeling can amplify early mistakes. If the model is confidently wrong, it will create bad labels and then learn from its own bad labels. So confidence thresholds, strong augmentation, and validation checks matter a lot.

Reinforcement learning

Reinforcement learning (RL) is about learning by interacting with an environment. Instead of a fixed dataset of input-output pairs, the agent takes actions, receives rewards, and changes the future data it will see.

The common formulation is a Markov Decision Process (MDP):

\[\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma)\]

where:

  • \(\mathcal{S}\) is the state space.
  • \(\mathcal{A}\) is the action space.
  • \(P(s' \mid s,a)\) is the transition probability.
  • \(R(s,a,s')\) is the reward.
  • \(\gamma \in [0,1)\) is the discount factor.

A policy \(\pi(a \mid s)\) tells the agent what action to take. The objective is to maximize expected discounted return:

\[J(\pi) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty} \gamma^t r_t \right]\]

The value function estimates how good a state is:

\[V^{\pi}(s) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty} \gamma^t r_t \mid s_0=s \right]\]

and the action-value function estimates how good it is to take action \(a\) in state \(s\):

\[Q^{\pi}(s,a) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty} \gamma^t r_t \mid s_0=s, a_0=a \right]\]

The Bellman equation is the core recursion:

\[V^{\pi}(s) = \sum_a \pi(a \mid s) \sum_{s'} P(s' \mid s,a) \left[R(s,a,s') + \gamma V^{\pi}(s')\right]\]

RL is powerful but also tricky because the data is not i.i.d., exploration matters, rewards can be sparse, and a poorly specified reward can produce weird behavior. The classic reference is Sutton and Barto’s Reinforcement Learning: An Introduction.

Machine Learning Settings

Representation learning

Representation learning is the problem of learning useful features automatically instead of hand-designing them.

Given raw input \(x\), we learn an encoder:

\[z = f_{\theta}(x)\]

where \(z\) should preserve the information needed for downstream tasks while removing irrelevant nuisance factors.

Examples:

  • In vision: learn features for edges, shapes, textures, objects, layouts.
  • In NLP: learn token/sentence/document embeddings.
  • In speech: learn acoustic representations that are robust to speaker and noise.

Good representations often have these properties:

  • Invariance: irrelevant changes should not move the representation too much. Example: a cat is still a cat after small crop/color changes.
  • Disentanglement: different factors of variation are separated. Example: identity vs pose vs lighting.
  • Transferability: a representation learned for one task is useful for another.
  • Compactness: unnecessary information is compressed away.

Personal note: representation learning is probably the most important idea behind modern deep learning. Most “model intelligence” is really “the model learned a useful internal coordinate system”.

Continual learning

Continual learning studies how a model can learn a sequence of tasks without forgetting old ones.

Suppose tasks arrive over time:

\[\mathcal{T}_1, \mathcal{T}_2, \ldots, \mathcal{T}_m\]

After training on \(\mathcal{T}_m\), we still want good performance on earlier tasks \(\mathcal{T}_1, \ldots, \mathcal{T}_{m-1}\).

The main problem is catastrophic forgetting: gradient updates for a new task overwrite parameters that were important for old tasks.

Common strategies:

  • Replay: keep some old data or generate old-like data and mix it with new data.
  • Regularization: penalize changes to parameters important for previous tasks, e.g., Elastic Weight Consolidation (EWC).
  • Dynamic architectures: add new modules/experts for new tasks.
  • Prompt/adapters: in modern foundation models, keep the base model mostly fixed and learn small task-specific components.

A simplified EWC-style objective:

\[\mathcal{L}_{new}(\theta) + \frac{\lambda}{2} \sum_i F_i(\theta_i - \theta_i^*)^2\]

where \(\theta^*\) are parameters after previous tasks and \(F_i\) estimates how important parameter \(i\) was.

The tradeoff is stability vs plasticity:

  • Stability: do not forget old knowledge.
  • Plasticity: still learn new knowledge.

This is one of those problems that sounds easy until you actually try it :D

Meta-learning

Meta-learning means “learning to learn”. Instead of training on one task, we train over a distribution of tasks so the model can adapt quickly to a new task.

Let tasks be sampled from \(p(\mathcal{T})\). Each task has a small support/train set and a query/test set:

\[\mathcal{D}_{\mathcal{T}}^{train}, \quad \mathcal{D}_{\mathcal{T}}^{test}\]

In optimization-based meta-learning, such as MAML, the model learns an initialization \(\theta\) that can adapt with a few gradient steps:

\[\theta'_{\mathcal{T}} = \theta - \alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}}^{train}(\theta)\]

Then the meta-objective optimizes the post-adaptation performance:

\[\min_{\theta} \sum_{\mathcal{T} \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}}^{test}(\theta'_{\mathcal{T}})\]

Types of meta-learning:

  • Metric-based: learn an embedding where nearest-neighbor classification works well.
  • Optimization-based: learn parameters that are easy to fine-tune.
  • Model-based: use memory or recurrence to adapt quickly.

Meta-learning is useful for few-shot learning, personalization, robotics, and any setup where we need fast adaptation from little data.

Transfer learning

Transfer learning uses knowledge learned from one task/domain to improve another task/domain.

The common deep learning recipe:

  1. Pretrain a model on a large source dataset/task.
  2. Reuse the encoder/backbone on a target task.
  3. Fine-tune all or part of the model on target data.

Mathematically:

\[z = f_{\theta}^{source}(x), \quad \hat{y} = g_{\phi}(z)\]

Then adapt \(\theta\) and/or \(\phi\) on the target dataset.

Examples:

  • ImageNet-pretrained CNN/ViT transferred to medical imaging.
  • Language model pretrained on web text transferred to summarization or classification.
  • Speech model pretrained on large audio transferred to speaker/emotion recognition.

When it works well, transfer learning reduces data requirements. When it works badly, we get negative transfer, where source knowledge hurts target performance.

💡 Practical rule: transfer works best when the low-level structure is shared. Natural images to natural images? Usually good. Natural images to medical scans? Maybe. Dogs to financial time series? Please do not make the poor model suffer.

Multi-task learning

Multi-task learning trains one model on multiple tasks at the same time, usually with a shared backbone and task-specific heads:

\[z = f_{\theta}(x), \quad \hat{y}^{(t)} = g_{\phi_t}(z)\]

The objective is a weighted sum of task losses:

\[\mathcal{L}(\theta, \{\phi_t\}) = \sum_{t=1}^{T} \lambda_t \mathcal{L}_t(\theta, \phi_t)\]

Examples:

  • A vision model predicts class, bounding box, depth, and segmentation mask.
  • An NLP model performs sentiment classification, NER, and entailment.
  • A recommender system predicts clicks, watch time, likes, and churn.

Why it can help:

  • Shared representation acts as regularization.
  • Related tasks provide extra supervision.
  • The model may learn features that a single task would not reveal.

Why it can fail:

  • Tasks may compete, causing negative transfer.
  • Loss scales can be very different.
  • One easy task can dominate the shared representation.

This is why task weighting is not just an implementation detail. It can decide whether multi-task learning is elegant or chaos wearing a lab coat.

Domain adaptation

Domain adaptation is transfer learning where the task is usually the same, but the data distribution changes between source and target domains.

Examples:

  • Train on synthetic images, test on real images.
  • Train on sunny driving scenes, test on rainy/night scenes.
  • Train a medical model in one hospital, deploy in another hospital.

Let source and target distributions be:

\[P_S(X,Y), \quad P_T(X,Y)\]

The goal is to learn a model with low target risk:

\[R_T(f) = \mathbb{E}_{(x,y) \sim P_T} \left[\ell(f(x), y)\right]\]

even when most labels come from \(P_S\).

Unsupervised domain adaptation

Given a set of \(M \geq 1\) source domains \(\{ \mathcal{D}_{S_m} \}_{m=1}^{M}\), each source domain is a collection of labeled examples:

\[\mathcal{D}_{S_m} = \{(x_j^{(m)}, y_j^{(m)})\}_{j=1}^{N_{S_m}}, \quad y_j^{(m)} \in [K] := \{1,2,\ldots,K\}\]

and one unlabeled target domain:

\[\mathcal{D}_T = \{x_j^T\}_{j=1}^{N_T}\]

The goal of unsupervised domain adaptation is to learn a classifier \(f_{\theta}\) that generalizes well to the target domain \(\mathcal{D}_T\), even though target labels are not available during training.

Example:

  • Source domain: Images of faces from different countries, each country can be considered as a domain with different geographic features.
  • Target domain: Images from a demographic group or capture condition that is not well represented in the labeled source domains.

Common methods:

  • Feature alignment: make source and target representations statistically similar.
  • Adversarial domain adaptation: train a domain classifier to distinguish source vs target, while the encoder tries to fool it. DANN is the classic example.
  • Pseudo-labeling: generate labels for target samples and retrain.
  • Image-to-image translation: translate source style to target style or vice versa.

Supervised domain adaptation

The setting is similar to unsupervised domain adaptation, but the target domain has labels:

\[\mathcal{D}_T = \{(x_j^T, y_j^T)\}_{j=1}^{N_T}\]

Usually \(N_T\) is much smaller than the source dataset, so the question becomes: how do we use source data without overwhelming the target signal?

Simple strategies:

  • Pretrain on source, fine-tune on target.
  • Mix source and target data with higher weight on target samples.
  • Learn domain-specific BatchNorm/adapters.
  • Use source data only for representation learning, then train the final classifier on target labels.

Challenges

  • Covariate shift: \(P_S(X) \neq P_T(X)\) but \(P(Y \mid X)\) is assumed similar.
  • Label shift: \(P_S(Y) \neq P_T(Y)\), e.g., class imbalance changes.
  • Concept shift: \(P_S(Y \mid X) \neq P_T(Y \mid X)\). This is the hard one because the meaning of features changes.
  • Negative transfer: adaptation makes target performance worse.
  • Target leakage: accidentally using target labels or validation choices in a way that makes evaluation too optimistic.
  • Pseudo-label confirmation bias: wrong target pseudo-labels reinforce themselves.

Domain generalization

Domain generalization is harder than domain adaptation because we do not get target-domain data during training.

Training:

\[\{ \mathcal{D}_{S_1}, \mathcal{D}_{S_2}, \ldots, \mathcal{D}_{S_M} \}\]

Testing:

\[\mathcal{D}_{T}, \quad T \notin \{S_1,\ldots,S_M\}\]

The goal is to learn features that survive across domains:

\[f_{\theta}: X \rightarrow Y\]

such that target risk is low for an unseen domain. A good survey is Generalizing to Unseen Domains.

Common approaches:

  • Domain-invariant representation learning: learn features shared by all source domains.
  • Data augmentation: create synthetic domain shifts during training.
  • Meta-learning: simulate train/test domain splits inside training.
  • Invariant risk minimization (IRM)-style ideas: encourage predictors that stay stable across environments.
  • Ensembling: combine models with different biases.

Important caveat: domain generalization is extremely evaluation-sensitive. If the validation domain is too similar to the test domain, performance may look better than it really is.

Optimization

Gradient descent

Gradient descent is the basic algorithm for minimizing a differentiable objective.

Given a loss function \(\mathcal{L}(\theta)\), we update parameters in the opposite direction of the gradient:

\[\theta_{t+1} = \theta_t - \eta \nabla_{\theta}\mathcal{L}(\theta_t)\]

where \(\eta\) is the learning rate.

Intuition:

  • The gradient points in the direction where the loss increases fastest.
  • We go in the negative gradient direction to reduce the loss.
  • The learning rate controls step size.

If \(\eta\) is too small, training is slow. If \(\eta\) is too large, the optimization can bounce around or diverge. This is why learning-rate schedules matter so much in practice.

Newton’s method

Newton’s method uses second-order information, meaning it looks at curvature through the Hessian matrix:

\[\theta_{t+1} = \theta_t - H^{-1} \nabla_{\theta}\mathcal{L}(\theta_t)\]

where:

\[H = \nabla_{\theta}^{2}\mathcal{L}(\theta)\]

For a quadratic function, Newton’s method can reach the optimum in one step. Very cool mathematically, but in deep learning it is usually too expensive because:

  • The Hessian can be huge.
  • Inverting the Hessian is expensive.
  • The Hessian may be indefinite, so the Newton direction is not always a descent direction.

Still, the idea is important because many optimizers approximate second-order behavior.

Fisher information

The Fisher information measures how sensitive a probability model is to changes in its parameters.

For a distribution \(p_{\theta}(x)\):

\[F(\theta) = \mathbb{E}_{x \sim p_{\theta}} \left[ \nabla_{\theta}\log p_{\theta}(x) \nabla_{\theta}\log p_{\theta}(x)^T \right]\]

Why it matters:

  • It appears in maximum likelihood estimation.
  • It defines the natural gradient direction.
  • It is used in continual learning methods such as EWC to estimate parameter importance.
  • It connects optimization and information geometry.

Natural gradient updates use \(F^{-1}\) instead of ordinary Euclidean geometry:

\[\theta_{t+1} = \theta_t - \eta F^{-1}\nabla_{\theta}\mathcal{L}(\theta_t)\]

The intuition is: do not take the biggest step in parameter space; take a good step in distribution space.

Hessian matrix

The Hessian matrix contains second derivatives:

\[H_{ij} = \frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j}\]

It describes local curvature:

  • Positive eigenvalues: locally convex direction.
  • Negative eigenvalues: locally concave direction.
  • Large eigenvalues: sharp curvature.
  • Small eigenvalues: flat direction.

Near a point \(\theta\), the loss can be approximated by a second-order Taylor expansion:

\[\mathcal{L}(\theta + \Delta) \approx \mathcal{L}(\theta) + \nabla \mathcal{L}(\theta)^T \Delta + \frac{1}{2}\Delta^T H \Delta\]

This is the mathematical root of many ideas: Newton’s method, curvature-aware optimization, sharpness, flat minima, Laplace approximation, and uncertainty estimation.

Stochastic gradient descent

Full-batch gradient descent computes the gradient over the entire dataset:

\[\nabla_{\theta}\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta}\ell_i(\theta)\]

Stochastic gradient descent (SGD) estimates it using a mini-batch \(\mathcal{B}\):

\[g_t = \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \nabla_{\theta}\ell_i(\theta_t)\] \[\theta_{t+1} = \theta_t - \eta g_t\]

Why SGD works well:

  • Mini-batches are much cheaper than full batches.
  • Gradient noise can help escape sharp or poor local regions.
  • It scales to massive datasets.

Momentum improves SGD by smoothing updates:

\[v_t = \beta v_{t-1} + g_t\] \[\theta_{t+1} = \theta_t - \eta v_t\]

The mental model: SGD is like walking downhill while being pushed around by noisy wind. Annoying, but sometimes that noise saves you from a bad valley.

Adam, RMSProp, AdaGrad

These optimizers adapt the learning rate per parameter.

AdaGrad

AdaGrad accumulates squared gradients:

\[r_t = r_{t-1} + g_t \odot g_t\] \[\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{r_t}+\epsilon} \odot g_t\]

It is useful for sparse features because rarely updated parameters get relatively larger steps. The downside is that \(r_t\) only grows, so the effective learning rate can become too small.

RMSProp

RMSProp fixes AdaGrad’s aggressive decay by using an exponential moving average:

\[r_t = \beta r_{t-1} + (1-\beta) g_t \odot g_t\] \[\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{r_t}+\epsilon} \odot g_t\]

Adam

Adam combines momentum and RMSProp-style adaptive scaling. It keeps first and second moment estimates:

\[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t\] \[v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2\]

Bias correction:

\[\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}\]

Update:

\[\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}\]

Adam is often the default because it is stable and forgiving. SGD with momentum can still generalize better in some vision setups, but Adam/AdamW is extremely common in transformers. The original Adam paper is Kingma and Ba, 2014.

Bayesian Optimization

Bayesian optimization is used when evaluating the objective is expensive, noisy, or derivative-free. Hyperparameter tuning is the classic ML example.

Instead of trying random settings blindly, Bayesian optimization builds a surrogate model of the objective:

\[f(x) \sim \mathcal{GP}(m(x), k(x,x'))\]

where \(x\) is a hyperparameter configuration and \(f(x)\) might be validation accuracy or validation loss.

The algorithm:

  1. Try a few configurations.
  2. Fit/update a probabilistic surrogate model.
  3. Use an acquisition function to choose the next configuration.
  4. Evaluate it.
  5. Repeat.

Common acquisition functions:

  • Expected improvement (EI): sample where improvement over the current best is expected.
  • Upper confidence bound (UCB): sample where the optimistic estimate is high.
  • Probability of improvement (PI): sample where improvement is likely.

The exploration-exploitation tradeoff is very explicit here:

  • Explore uncertain regions.
  • Exploit regions likely to be good.

Bayesian optimization is great when each experiment is expensive, but less useful when cheap random/grid search is already enough. See Snoek et al. and Frazier’s tutorial for deeper reading.

Bias-Variance

Bias – Error due to overly simplistic assumptions in the model.

  • High bias models underfit the data because they fail to capture important patterns.
  • Example: A linear regression model trying to fit a complex, nonlinear dataset.

Variance – Error due to excessive sensitivity to training data.

  • High variance models overfit the data by learning noise along with patterns.
  • Example: A deep neural network memorizing training data instead of generalizing.

It is worth noting that bias and variance are theoretical quantities over possible training sets and the true data-generating distribution. In practice, we approximate this idea using validation/test performance, not training performance.

\[\text{Bias} = \mathbb{E}[\hat{f}(x)] - f(x)\]
  • high bias means large errors due to incorrect assumptions or model is too simple
  • low bias means small errors, and close to the ground truth
\[\text{Variance} = \mathbb{E}[\hat{f}(x) - \mathbb{E}[\hat{f}(x)]]^2\]
  • high variance means the model makes different predictions each time after training on different datasets.
  • low variance means similar predictions on different datasets

Tradeoff

  • A model with high bias and low variance is simple but inaccurate (underfitting).
  • A model with low bias and high variance is flexible but unreliable (overfitting).
  • The goal is to find a balance—an optimal point where bias and variance are minimized to achieve good generalization.

How to Control Bias-Variance Tradeoff?

  • Increase model complexity to reduce bias (e.g., moving from linear to polynomial models).
  • Reduce complexity to lower variance (e.g., regularization, pruning in decision trees).
  • Use more training data to help models generalize better.
  • Use ensemble methods like bagging and boosting to balance bias and variance.

Bagging and Boosting

Bagging and Boosting

Reference: https://towardsdatascience.com/ensemble-learning-bagging-boosting-3098079e5422

Bagging (Bootstrap Aggregating)

💡 Goal: Reduce variance and prevent overfitting by training multiple models in parallel and averaging their predictions.

How it works?

  • Bootstrap Sampling: Random subsets of training data (with replacement) are drawn.
  • Train Weak Learners: Independent models (often decision trees) are trained on these subsets.
  • Aggregate Predictions:
    • For regression: Take the average of predictions.
    • For classification: Use majority voting.

Boosting

💡 Goal: Reduce bias and improve weak models by training them sequentially, where each new model focuses on the errors of the previous one.

How it works?

  • Train a Weak Model (e.g., a shallow decision tree).
  • Identify Errors: Assign higher weights to misclassified instances.
  • Train the Next Model: Focus more on difficult cases.
  • Repeat: Continue training models sequentially, combining them for the final prediction.

Popular Boosting Algorithms

🔥 AdaBoost (Adaptive Boosting)

  • Adjusts sample weights to focus on hard-to-classify instances.
  • Final prediction is a weighted sum of weak models.

🔥 Gradient Boosting (e.g., XGBoost, LightGBM, CatBoost)

  • Models are trained sequentially to minimize the residual errors of previous models.
  • Uses gradient descent to optimize performance.

🔥 Boosted Trees (e.g., XGBoost)

  • A more efficient version of gradient boosting that includes regularization.
  • Popular for Kaggle competitions and real-world ML tasks.

XGBoost

XGBoost stands for Extreme Gradient Boosting. It is an efficient and regularized implementation of gradient boosted decision trees, introduced in XGBoost: A Scalable Tree Boosting System.

The model is an additive ensemble of trees:

\[\hat{y}_i = \sum_{t=1}^{T} f_t(x_i), \quad f_t \in \mathcal{F}\]

where each \(f_t\) is a decision tree.

At each boosting step, XGBoost adds a new tree to reduce the objective:

\[\mathcal{L}^{(t)} = \sum_{i=1}^{N} \ell(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega(f_t)\]

The regularization term penalizes tree complexity:

\[\Omega(f) = \gamma T + \frac{1}{2}\lambda \sum_{j=1}^{T} w_j^2\]

where \(T\) is the number of leaves and \(w_j\) is the score of leaf \(j\).

Why XGBoost is strong:

  • Uses first-order and second-order gradients.
  • Has explicit regularization.
  • Handles sparse features well.
  • Supports shrinkage/learning rate, column subsampling, and row subsampling.
  • Is highly optimized for speed and memory.

Parameters people tune a lot:

  • n_estimators: number of trees.
  • max_depth: depth of each tree.
  • learning_rate: shrinkage factor.
  • subsample: fraction of rows used per tree.
  • colsample_bytree: fraction of features used per tree.
  • lambda and alpha: L2 and L1 regularization.

Practical note: XGBoost is still extremely competitive for tabular data. Deep learning wins many image/text/audio tasks, but for structured tables, boosted trees are often still the “please do not overcomplicate this” baseline.

K-Nearest Neighbors (KNN)

1️⃣ Choose a value for K (number of neighbors). 2️⃣ Compute the distance between the new data point and all existing points in the dataset (e.g., Euclidean distance). 3️⃣ Select the K closest neighbors. 4️⃣ Make a prediction:

  • Classification: Assign the most common class among neighbors (majority voting).
  • Regression: Take the average of the target values of the K neighbors.

Choose K:

  • Small K (e.g., K=1 or 3):
    • ✅ Captures details but can overfit (high variance).
  • Large K (e.g., K=10 or 20):
    • ✅ More generalization but may underfit (high bias).
  • Common practice: Choose K using cross-validation (often √N, where N is the dataset size).

K-Means Clustering

K-means clustering is a popular unsupervised machine learning algorithm that is used to partition a dataset into k clusters. The algorithm works by iteratively assigning instances to the nearest cluster center and updating the cluster centers based on the mean of the instances in each cluster. The goal of k-means clustering is to minimize the sum of squared distances between instances and their respective cluster centers.

Pseudocode for k-means clustering:

  1. Initialize k cluster centers randomly or using a heuristic.
  2. Assign each instance to the nearest cluster center.
  3. Update the cluster centers based on the mean of the instances in each cluster.
  4. Repeat steps 2 and 3 until convergence.
  5. Return the final cluster assignments and cluster centers.

K-means clustering is sensitive to the initial cluster centers and can converge to a local minimum. To mitigate this issue, the algorithm is often run multiple times with different initializations, and the best clustering is selected based on a predefined criterion.

Applications of k-means clustering include customer segmentation, image compression, and anomaly detection.

Logistic regression

Sigmoid function:

\[\sigma(z) = \frac{1}{1 + e^{-z}}\]

where \(z = wX + b\) is the linear combination of the input features and the model parameters. The nice thing about the sigmoid function is that it maps any real-valued number to the range [0, 1], which is very useful for binary classification.

Cost function:

\[\mathcal{L}(\theta) = - \frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log \hat{y}_i + (1 - y_i) \log (1 - \hat{y}_i) \right]\]

where \(\hat{y}_i = \sigma(z_i)\) is the predicted probability of the \(i\)-th sample.

Gradient descent:

\[\nabla_{\theta} \mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)X_i\]

Then the update is \(\theta \leftarrow \theta - \eta \nabla_{\theta}\mathcal{L}(\theta)\).

Bayesian theorem

Bayesian theorem:

\[P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}\]

Where \(A\) and \(B\) are events, e.g., A: having a cancer, B: having a positive test result, and \(P(A \mid B)\) is the probability of \(A\) given \(B\). Bayesian theorem can be used to update our belief based on new evidence (given \(B\))

  • \(P(A \mid B)\) is the posterior probability of \(A\) given \(B\).
  • \(P(B \mid A)\) is the likelihood of \(B\) given \(A\). How likely a patient with cancer (\(A\)) has a positive test result (\(B\)).
  • \(P(A)\) is the prior probability of \(A\). The probability of having cancer over all the population.
  • \(P(B)\) is the marginal probability of \(B\). The probability of having a positive test result over all the population, including patients with and without cancer.

The marginal probability \(P(B)\) can be computed by summing over all the possible values of \(A\) and usually very expensive to compute because it requires summing over all the possible values of \(A\).

\[P(B) = \sum_{A} P(B \mid A)P(A)\]

What is the Naive Bayes classifier?

The Naive Bayes classifier is a simple and efficient machine learning algorithm that is based on Bayes’ theorem and the assumption of conditional independence between features. Despite its simplicity, the Naive Bayes classifier is often used as a baseline model for text classification and other tasks.

The Naive Bayes classifier is particularly well-suited for text classification tasks, such as spam detection and sentiment analysis, where the input features are typically word frequencies or presence/absence of words.

Mathematically, the Naive Bayes classifier predicts the class label of an instance based on the maximum a posteriori (MAP) estimation:

\[\hat{y} = \underset{y \in \mathcal{Y}}{\text{argmax}} P(y|X)\]

Where:

  • \(\hat{y}\) is the predicted class label.
  • \(\mathcal{Y}\) is the set of possible class labels.
  • \(X\) is the input features.
  • \(P(y \mid X)\) is the posterior probability of class label y given the input features X. The posterior probability \(P(y \mid X)\) is calculated using Bayes’ theorem and the assumption of conditional independence, i.e., \(P(y \mid X) = \frac{P(X \mid y)P(y)}{P(X)}\). The denominator \(P(X)\) is constant for all class labels and can be ignored for the purpose of classification.

With the Naive Bayes assumption:

\[P(X \mid y) = \prod_{j=1}^{d} P(x_j \mid y)\]

So classification becomes:

\[\hat{y} = \underset{y \in \mathcal{Y}}{\text{argmax}} \ P(y)\prod_{j=1}^{d}P(x_j \mid y)\]

Curse of dimensionality

The curse of dimensionality refers to the challenges and problems that arise when working with high-dimensional data. As the number of features (dimensions) increases, data becomes sparse, distances become less meaningful, and models struggle to generalize well.

And because distance becomes less meaningful, algorithms that rely on distance such as KNN, SVM, or K-means become less effective.

And because data becomes sparse (but the true pattern is still in lower dimensions), machine learning models need more data to fill in the space, otherwise they will overfit the data.

Overcoming overfitting

L1 vs L2 regularization

Regularization helps prevent overfitting by adding a penalty to the loss function, discouraging overly complex models (large weights or many features used).

L1 regularization:

\[\mathcal{L}(\theta) + \lambda \sum_{i=1}^{n} |\theta_i|\]

L2 regularization:

\[\mathcal{L}(\theta) + \lambda \sum_{i=1}^{n} \theta_i^2\]

L1 regularization encourages sparsity in the model’s weights (some weights will be exactly 0), while L2 regularization encourages small weights (no weight will be exactly 0 but very small).

Because L1 regularization encourages sparsity, it can be used for feature selection (eliminate some features that are not important - set their weights to 0). But it can lead to unstable solutions when features are highly correlated/entangled.

Elastic net regularization is a combination of L1 and L2 regularization.

\[\mathcal{L}(\theta) + \lambda_1 \sum_{i=1}^{n} |\theta_i| + \lambda_2 \sum_{i=1}^{n} \theta_i^2\]

Dropout

Dropout is a regularization technique used in neural networks to prevent overfitting by randomly “dropping” (deactivating) neurons during training. This forces the network to learn more robust and generalizable features rather than relying on specific neurons.

🔹 Why? In deep networks, some neurons may become too dependent on others, leading to overfitting. Works well in fully connected layers, less useful in convolutional layers (which have spatial dependencies and less neurons than fully connected layers).

🔹 How? During training, for each mini-batch, some neurons are randomly dropped out with a probability \(p\) (e.g., 0.5). Important thing that most people forget: the remaining neurons need to be scaled up by \(\frac{1}{1-p}\) to keep the expected output the same.

Early stopping

Early stopping is a regularization technique that stops training when the model’s performance on the validation set starts to degrade (model starts to overfit or memorize the training data - therefore, the performance on the validation set starts to decrease).

Batch normalization/layer normalization

Batch Normalization

Batch Normalization normalizes activations across the batch dimension for each feature. The original paper is Ioffe and Szegedy, 2015.

\[\mu = \frac{1}{m} \sum_{i=1}^{m} x_i\] \[\sigma^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu)^2\] \[\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}\] \[y_i = \gamma \hat{x}_i + \beta\]

where \(\gamma\) and \(\beta\) are learnable scale and shift parameters.

This transformation usually makes optimization more stable. The original motivation was reducing internal covariate shift, although later work also points to smoother optimization geometry as a big reason BatchNorm works well.

However, batch statistics vary across mini-batches, making it problematic for small batch sizes or RNNs. Therefore, batch normalization is commonly used in CNNs and Fully Connected Networks (FCNs). In inference, we use the running mean and variance to normalize the activations.

Internal Covariate Shift

Internal Covariate Shift occurs when the distribution of activations (features) changes during training due to updates in previous layers. This makes optimization harder because each layer must constantly adapt to new distributions of inputs. It causes slow convergence and unstable training.

It differs from the covariate shift problem, which is the change in the distribution of the input features (e.g., training on sunny images but testing on rainy ones). Internal Covariate Shift happens inside a deep network during training.

Layer Normalization

Layer Normalization normalizes activations across features for each individual data point (sample-wise normalization). See Layer Normalization.

For a vector \(x \in \mathbb{R}^{d}\):

\[\mu = \frac{1}{d}\sum_{j=1}^{d}x_j\] \[\sigma^2 = \frac{1}{d}\sum_{j=1}^{d}(x_j-\mu)^2\] \[\hat{x}_j = \frac{x_j-\mu}{\sqrt{\sigma^2+\epsilon}}\] \[y_j = \gamma_j \hat{x}_j + \beta_j\]

Key Properties:

  • Works sample-wise (statistics are computed for each input independently).
  • Works well for RNNs and small batch sizes, unlike BatchNorm.
  • More stable training in NLP and reinforcement learning tasks.
  • Less effective than BatchNorm for CNNs, where spatial dependencies are important.

Data augmentation

Data augmentation creates new training examples by applying label-preserving transformations.

For image classification:

  • Random crop
  • Horizontal flip
  • Color jitter
  • Rotation
  • Blur
  • Cutout / random erasing
  • MixUp / CutMix

For text:

  • Back-translation
  • Token masking
  • Paraphrasing
  • Random deletion/swapping (careful, this can change meaning)

For audio:

  • Time shift
  • Noise injection
  • SpecAugment-style time/frequency masking

Mathematically, if \(t \sim \mathcal{T}\) is a random augmentation:

\[x' = t(x), \quad y' = y\]

Then we train on \(\{(x',y)\}\).

Why it helps:

  • Increases effective dataset size.
  • Encourages invariance to nuisance factors.
  • Reduces overfitting.
  • Makes the model more robust to real-world variation.

Important caveat: augmentation must preserve the label. Rotating a digit “6” into “9” is not harmless. Flipping medical images can be problematic if left/right anatomy matters. This is one of those details that looks small until it ruins the experiment.

Cross-validation

Cross-validation estimates generalization by splitting the data into multiple train/validation folds. The common version is \(k\)-fold cross-validation:

  1. Split the dataset into \(k\) folds.
  2. Train on \(k-1\) folds.
  3. Validate on the remaining fold.
  4. Repeat so each fold is used once as validation.
  5. Average the validation metrics.

The estimate is:

\[\mathrm{CV} = \frac{1}{k}\sum_{j=1}^{k} \mathrm{Score}_j\]

Common variants:

  • Stratified k-fold: preserves class proportions.
  • Leave-one-out: \(k=N\); expensive but data-efficient.
  • Group k-fold: keeps related samples in the same fold to avoid leakage.
  • Time-series split: respects temporal ordering.

💡 Very important: the test set should not be used for model selection. Use train/validation or cross-validation for choosing hyperparameters, then report final performance once on the test set. Otherwise, the test set slowly becomes training data wearing a disguise.

Generalization

Generalization is the ability of a model to perform well on unseen data, not just the training data.

The generalization gap is:

\[\mathrm{Gap} = \mathcal{L}_{test}(\theta) - \mathcal{L}_{train}(\theta)\]

Small training loss alone is not enough. Modern deep networks can even fit random labels, which is one reason generalization in deep learning is still theoretically interesting (Zhang et al., 2016).

Things that usually help generalization:

  • More diverse data
  • Better inductive bias
  • Regularization
  • Data augmentation
  • Early stopping
  • Ensembling
  • Pretraining
  • Better validation protocol

Things that hurt generalization:

  • Data leakage
  • Overfitting hyperparameters to the validation/test set
  • Spurious correlations
  • Distribution shift
  • Label noise
  • Train/test preprocessing mismatch

Out-of-distribution generalization

Out-of-distribution (OOD) generalization asks whether a model trained on one distribution can work on another:

\[P_{train}(X,Y) \neq P_{test}(X,Y)\]

Examples:

  • Train on hospital A, test on hospital B.
  • Train on clean images, test on corrupted images.
  • Train on one country/language/style, test on another.
  • Train on historical data, test after the world changes.

Types of shift:

  • Covariate shift: \(P(X)\) changes but \(P(Y \mid X)\) is stable.
  • Label shift: \(P(Y)\) changes.
  • Concept shift: \(P(Y \mid X)\) changes.
  • Spurious correlation shift: shortcuts learned during training break at test time.

OOD generalization is hard because empirical risk minimization assumes train and test are sampled from the same distribution:

\[P_{train} = P_{test}\]

When this assumption breaks, validation accuracy can lie to us. A model may look excellent on i.i.d. validation data and then fail in deployment.

Practical tools:

  • Evaluate on multiple domains, not one random split.
  • Use group splits if samples are correlated.
  • Stress-test with corruptions/perturbations.
  • Track calibration, not just accuracy.
  • Prefer features that are causally related to the label when possible.

Adversarial generalization/training

Adversarial examples are inputs that look almost unchanged to humans but cause a model to make a wrong prediction. The classic paper is Explaining and Harnessing Adversarial Examples.

For an input \(x\) and label \(y\), FGSM creates an adversarial example:

\[x_{adv} = x + \epsilon \cdot \mathrm{sign} \left( \nabla_x \mathcal{L}(\theta, x, y) \right)\]

The gradient is taken with respect to the input, not the model weights. This is the key idea.

Adversarial training solves a robust optimization problem:

\[\min_{\theta} \mathbb{E}_{(x,y)\sim \mathcal{D}} \left[ \max_{\|\delta\| \leq \epsilon} \mathcal{L}(\theta, x+\delta, y) \right]\]

Inner loop: find a perturbation \(\delta\) that makes the model fail.
Outer loop: update the model so it survives that perturbation.

Why it matters:

  • Improves robustness to worst-case perturbations.
  • Reveals that high accuracy does not imply robust understanding.
  • Often creates a tradeoff between clean accuracy and adversarial robustness.

Personal note: adversarial examples are a humbling reminder that neural networks and humans do not always use the same evidence, even when they produce the same label.

Sharpness-aware minimization

Sharpness-aware minimization (SAM) tries to find parameters that are not only low-loss, but also live in a flat neighborhood of low loss. See Foret et al., 2020.

Standard training:

\[\min_w \mathcal{L}(w)\]

SAM:

\[\min_w \max_{\|\epsilon\|_2 \leq \rho} \mathcal{L}(w+\epsilon)\]

The inner maximization asks: “if I perturb the weights a little, how bad can the loss get?”

The perturbation is often approximated by:

\[\epsilon(w) = \rho \frac{\nabla_w \mathcal{L}(w)} {\|\nabla_w \mathcal{L}(w)\|_2}\]

Then the model updates using the gradient at the perturbed weights:

\[\nabla_w \mathcal{L}(w+\epsilon(w))\]

Why this can help:

  • Sharp minima can be sensitive to small changes.
  • Flat minima often correlate with better generalization.
  • SAM explicitly discourages solutions where the loss increases quickly around the learned weights.

Cost: SAM usually needs two gradient computations per step, so it is slower than ordinary training.

Evaluation

Metrics

Type I and Type II error

Type I and Type II error

Null hypothesis (H0) is a statement that there is no relationship between two measured phenomena, or no association among groups. It is the default assumption that there is no effect or no difference. The alternative hypothesis (H1) is the statement that there is a relationship between two measured phenomena, or an association among groups. It is the opposite of the null hypothesis.

In the example of Innocent and Guilty, the null hypothesis is “Innocent” and the alternative hypothesis is “Guilty”. Type I error is the incorrect rejection of the null hypothesis (false positive), while Type II error is the failure to reject the null hypothesis when it is false (false negative).

ROC curve

ROC stands for Receiver Operating Characteristic. It plots:

\[\mathrm{TPR} = \frac{TP}{TP + FN}\]

against:

\[\mathrm{FPR} = \frac{FP}{FP + TN}\]

as we vary the classification threshold.

Where:

  • TPR is recall/sensitivity.
  • FPR is the false positive rate.

ROC is useful when we care about ranking positives above negatives across many possible thresholds.

Important caveat: ROC curves can look overly optimistic when the positive class is very rare. In imbalanced settings, precision-recall curves are often more informative.

Precision-recall curve

Precision:

\[\mathrm{Precision} = \frac{TP}{TP + FP}\]

Recall:

\[\mathrm{Recall} = \frac{TP}{TP + FN}\]

The precision-recall curve shows the tradeoff between precision and recall as the decision threshold changes.

Use it when:

  • The positive class is rare.
  • False positives are expensive.
  • You care about the quality of positive predictions.

Example: disease detection, fraud detection, dangerous-content detection, anomaly detection.

F1 score

F1 is the harmonic mean of precision and recall:

\[F1 = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}} {\mathrm{Precision} + \mathrm{Recall}}\]

It is high only when both precision and recall are high.

Why harmonic mean? Because it punishes imbalance. If precision is 1.0 but recall is 0.0, F1 is still 0.0.

Use F1 when:

  • You need one summary number.
  • Precision and recall are both important.
  • Classes are imbalanced.

Do not use F1 blindly if false positives and false negatives have very different costs. In that case, choose a metric aligned with the real decision cost.

AUC

AUC means Area Under the Curve.

Common forms:

  • ROC-AUC: area under the ROC curve.
  • PR-AUC / Average Precision: area-like summary under the precision-recall curve.

ROC-AUC has a nice probabilistic interpretation:

\[\mathrm{AUC} = P(s(x^+) > s(x^-))\]

where \(x^+\) is a randomly chosen positive sample, \(x^-\) is a randomly chosen negative sample, and \(s(\cdot)\) is the model score.

Interpretation:

  • 0.5: random ranking.
  • 1.0: perfect ranking.
  • Below 0.5: worse than random, probably something is flipped.

Important caveat: AUC measures ranking quality, not calibration. A model can have high AUC but terrible probabilities. If probabilities matter, also check calibration metrics like Brier score or reliability diagrams.

References