Machine Learning (ML) is a branch of Artificial Intelligence (AI) concerned with building systems that learn from data without being explicitly programmed. Rather than following hard-coded rules, ML algorithms identify patterns and relationships within data to make predictions or decisions.
The field is broadly divided into three main paradigms:
Supervised learning consists of learning a mapping \(f: X \rightarrow Y\) from a set of labeled training examples \(\{(x_i, y_i)\}_{i=1}^{n}\), where \(x_i \in X\) is an input (feature vector) and \(y_i \in Y\) is the corresponding output (label or target).
The goal is to find the function \(f\) that minimizes the expected loss:
\[ \min_f \; \mathbb{E}[L(Y, f(X))] \]
where \(L\) is a loss function measuring the discrepancy between the true value \(Y\) and the prediction \(f(X)\).
In classification, the target variable \(Y\) takes values in a finite set of class labels, e.g. \(Y \in \{0, 1, \ldots, K-1\}\). The goal is to assign each input to the correct class.
Common loss functions include:
Examples: spam detection (spam vs. not spam), image classification (cat, dog, bird, …), medical diagnosis (disease present vs. absent).
In regression, the target variable \(Y\) is continuous (real-valued). The goal is to predict a numerical quantity as accurately as possible.
The most common loss function is the Mean Squared Error (MSE):
\[ \text{MSE} = \mathbb{E}\left[(Y - f(X))^2\right] \]
Examples: predicting house prices, stock price forecasting, estimating patient recovery time.
The available data is split into a training set (used to fit the model) and a test set (used to evaluate generalization performance). A typical split is 70-80% training and 20-30% test.
The expected prediction error of an estimator \(\hat{f}\) can be decomposed as:
\[ \mathbb{E}\left[(Y - \hat{f}(X))^2\right] = \text{Bias}^2(\hat{f}) + \text{Var}(\hat{f}) + \sigma^2 \]
where:
The dataset is partitioned into \(k\) equally-sized folds. The model is trained on \(k-1\) folds and validated on the remaining fold, rotating \(k\) times. The final performance estimate is the average across all folds:
\[ \text{CV}(k) = \frac{1}{k}\sum_{i=1}^{k} L_i \]
Regularization adds a penalty term to the loss function to discourage overly complex models:
\[ \min_f \; \mathbb{E}[L(Y, f(X))] + \lambda \, \Omega(f) \]
where \(\lambda > 0\) controls the strength of regularization and \(\Omega(f)\) penalizes complexity (e.g. \(\|\beta\|_2^2\) for Ridge, \(\|\beta\|_1\) for Lasso).
In unsupervised learning there are no labels. The algorithm receives only input data \(\{x_i\}_{i=1}^{n}\) and must discover hidden structure, patterns, groups, or compact representations within the data.
The goal is to partition data points into groups (clusters) such that points within the same cluster are more similar to each other than to points in other clusters.
The goal is to represent high-dimensional data in a lower-dimensional space while preserving as much information as possible.
Discovers interesting relationships between variables in large datasets. Classical application: market basket analysis (e.g. "customers who buy bread also tend to buy butter"). Key metrics include support, confidence, and lift.
Estimates the underlying probability density function \(p(x)\) from observed data. Methods include kernel density estimation (KDE) and Gaussian mixture models (GMM):
\[ \hat{p}(x) = \frac{1}{nh}\sum_{i=1}^{n} K\left(\frac{x - x_i}{h}\right) \]
where \(K\) is a kernel function and \(h\) is the bandwidth.
Combines a small amount of labeled data with a large amount of unlabeled data. The idea is that the structure of the unlabeled data (clusters, manifolds) provides useful information for learning. This is especially valuable when labeling is expensive or time-consuming.
An agent interacts with an environment, taking actions and receiving rewards (or penalties). The goal is to learn a policy \(\pi(a|s)\) that maximizes the expected cumulative reward:
\[ \max_\pi \; \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t \, r_t\right] \]
where \(\gamma \in [0,1)\) is a discount factor and \(r_t\) is the reward at time step \(t\). Applications include game playing, robotics, and autonomous driving.
A simple decision flowchart for selecting the appropriate learning paradigm:
Do you have labeled data?
│
├── YES → Supervised Learning
│ │
│ ├── Is Y continuous? → Regression
│ │
│ └── Is Y discrete? → Classification
│
└── NO → Unsupervised Learning
│
├── Want to find groups? → Clustering
│
└── Want to reduce dimensions? → Dimensionality Reduction
In practice, the choice also depends on the amount of available data, computational resources, interpretability requirements, and the specific problem domain.