Loss Functions: Complete Notes (Beginner to Advanced)

---

1. Loss Function#

A loss function measures how far off a single prediction is from the actual true value. It takes the model's predicted output and the true target as input, and returns a single number representing the "error" or "cost" of that prediction.

loss = L(y_true, y_predicted)

• A low loss value means the prediction is close to the true value.
• A high loss value means the prediction is far from the true value.
• The loss is calculated per data point (or per batch, then usually averaged).

Purpose in training: the loss function's output is what backpropagation uses to compute gradients. The gradients tell each weight and bias how much and in which direction they should change to reduce the loss.

Without a loss function: the network has no numerical signal telling it whether a prediction is good or bad, so there is nothing to differentiate and nothing to guide weight updates. Training would not be possible at all.

With a loss function: the network has a clear, differentiable target to minimize, which is the mathematical foundation of the entire training process.

Code example (basic loss computation):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def mean_squared_error(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

y_true = np.array([3.0, 5.0, 2.5])
y_pred = np.array([2.8, 5.2, 2.0])

loss = mean_squared_error(y_true, y_pred)
print("Loss:", loss)

---

2. Objective Function#

The objective function is the broader, general term for the function that an optimization algorithm is trying to minimize (or maximize). In the context of neural networks, this is usually the average loss across the entire training dataset (or a batch), sometimes combined with additional terms like regularization penalties.

Objective = (1/N) * sum(loss(y_true_i, y_pred_i) for i in 1..N) + regularization_term

Distinction between "loss function" and "objective function":

In everyday practice and most deep learning frameworks, these three terms (loss, cost, objective) are frequently used interchangeably, since the regularization term is often optional or already merged into how the loss is defined for that specific problem.

Without regularization added to the objective function: the model optimizes purely to minimize prediction error on the training set, which can lead it to memorize the training data too closely (overfitting).

With regularization terms added into the objective function: the model is penalized for having overly large or complex weights, encouraging it to find simpler, more generalizable solutions.

Code example (objective function with L2 regularization added):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

def l2_regularization(weights, lambda_reg=0.01):
    return lambda_reg * np.sum(weights ** 2)

y_true = np.array([3.0, 5.0, 2.5])
y_pred = np.array([2.8, 5.2, 2.0])
weights = np.array([0.5, -0.3, 0.8])

data_loss = mse_loss(y_true, y_pred)
reg_term = l2_regularization(weights)
objective = data_loss + reg_term

print("Data loss:", data_loss)
print("Regularization term:", reg_term)
print("Total objective:", objective)

---

3. Regression Losses#

Regression losses are used when the model's target is a continuous numeric value (e.g., predicting house prices, temperature, or a stock value), rather than a discrete category.

3.1 MSE (Mean Squared Error)#

MSE = (1/N) * sum((y_true_i - y_pred_i)^2 for i in 1..N)

• Squares the error for each prediction, then averages across all samples.
• Because errors are squared, larger errors are penalized much more heavily than smaller ones (an error of 4 contributes 16 to the sum, while an error of 1 contributes only 1).
• The squaring also makes MSE differentiable everywhere, which is convenient for gradient-based optimization.

Without squaring the error (using raw differences): positive and negative errors could cancel each other out when summed/averaged, giving a misleadingly low error even when individual predictions are far off.

With MSE: all errors become positive (since they're squared) before averaging, and large mistakes are punished disproportionately, pushing the model to avoid big misses.

Sensitivity to outliers: because of the squaring, MSE is very sensitive to outliers. A single very wrong prediction can dominate the entire loss value.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

y_true = np.array([3.0, 5.0, 2.5, 10.0])
y_pred = np.array([2.8, 5.2, 2.0, 2.0])  # last prediction is a big outlier error

print("MSE:", mse(y_true, y_pred))

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using PyTorch
import torch
import torch.nn as nn

loss_fn = nn.MSELoss()
y_true = torch.tensor([3.0, 5.0, 2.5])
y_pred = torch.tensor([2.8, 5.2, 2.0])
print("MSE (PyTorch):", loss_fn(y_pred, y_true).item())

3.2 MAE (Mean Absolute Error)#

MAE = (1/N) * sum(|y_true_i - y_pred_i| for i in 1..N)

• Takes the absolute value of the error for each sample, then averages.
• Unlike MSE, all errors are weighted linearly rather than quadratically, so large errors are not disproportionately punished.
• More robust to outliers than MSE, because a single large error contributes proportionally to its size rather than its square.

Without using absolute value (using MSE instead when outliers are a concern): a few extreme outliers in your data could dominate the training signal, causing the model to overcorrect for those rare cases at the expense of overall accuracy on typical data.

With MAE: outliers still contribute to the loss, but not disproportionately more than their actual magnitude, giving a more balanced training signal when your dataset has noisy or extreme values.

Trade-off: MAE's gradient is constant (either +1 or -1) regardless of how far the prediction is from the target, which can make optimization near the minimum less smooth compared to MSE, whose gradient naturally shrinks as the error gets smaller.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

y_true = np.array([3.0, 5.0, 2.5, 10.0])
y_pred = np.array([2.8, 5.2, 2.0, 2.0])

print("MAE:", mae(y_true, y_pred))

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using PyTorch
import torch
import torch.nn as nn

loss_fn = nn.L1Loss()  # MAE is called L1Loss in PyTorch
y_true = torch.tensor([3.0, 5.0, 2.5])
y_pred = torch.tensor([2.8, 5.2, 2.0])
print("MAE (PyTorch):", loss_fn(y_pred, y_true).item())

3.3 Huber Loss#

Huber Loss combines the best properties of both MSE and MAE. It behaves like MSE for small errors (smooth, well-behaved gradient) and like MAE for large errors (robust to outliers).

Mathematical Formulation
Copy
Huber(y_true, y_pred) =
    0.5 * (y_true - y_pred)^2                     if |y_true - y_pred| <= delta
    delta * (|y_true - y_pred| - 0.5 * delta)     if |y_true - y_pred| > delta

Where delta is a hyperparameter that controls the threshold at which the loss transitions from quadratic (MSE-like) to linear (MAE-like) behavior.

Without Huber loss (choosing purely between MSE or MAE): you are forced to either accept MSE's oversensitivity to outliers or MAE's less smooth gradient near the minimum. You cannot get both benefits with a single one of these two losses alone.

With Huber loss: you get the smooth, stable gradient behavior of MSE for typical, well-behaved errors, combined with the outlier-robustness of MAE for extreme errors, making it a popular default choice for regression problems with noisy data.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def huber_loss(y_true, y_pred, delta=1.0):
    error = y_true - y_pred
    is_small_error = np.abs(error) <= delta
    squared_loss = 0.5 * error ** 2
    linear_loss = delta * (np.abs(error) - 0.5 * delta)
    return np.mean(np.where(is_small_error, squared_loss, linear_loss))

y_true = np.array([3.0, 5.0, 2.5, 10.0])
y_pred = np.array([2.8, 5.2, 2.0, 2.0])

print("Huber Loss:", huber_loss(y_true, y_pred, delta=1.0))

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using PyTorch
import torch
import torch.nn as nn

loss_fn = nn.HuberLoss(delta=1.0)
y_true = torch.tensor([3.0, 5.0, 2.5, 10.0])
y_pred = torch.tensor([2.8, 5.2, 2.0, 2.0])
print("Huber Loss (PyTorch):", loss_fn(y_pred, y_true).item())

Regression Losses Comparison#

---

4. Classification Losses#

Classification losses are used when the model's target is a discrete category (e.g., spam vs. not spam, or which of 10 digits an image represents), rather than a continuous number. Most classification losses are variations of cross-entropy, which measures the difference between two probability distributions: the true distribution (actual labels) and the predicted distribution (model output probabilities).

4.0 Cross-Entropy (General Concept)#

Cross-entropy comes from information theory. It measures how well a predicted probability distribution q matches a true probability distribution p.

CrossEntropy(p, q) = -sum(p_i * log(q_i) for all classes i)

• If the predicted probability for the correct class is close to 1, the loss is close to 0 (good).
• If the predicted probability for the correct class is close to 0, the loss becomes very large (heavily penalized), since -log(x) approaches infinity as x approaches 0.

Without cross-entropy (e.g., trying to use MSE for classification): MSE treats the difference between probabilities linearly/quadratically and does not heavily punish confidently wrong predictions the way cross-entropy does. This produces weaker gradient signals and generally slower, less effective training for classification tasks.

With cross-entropy: the loss sharply penalizes confident wrong predictions and rewards confident correct predictions, aligning well with how probability-based classification should be evaluated and optimized.

All three classification losses below (Binary Cross-Entropy, Categorical Cross-Entropy, and Sparse Categorical Cross-Entropy) are specific applications of this same cross-entropy concept, differing only in how the labels are formatted and how many classes are involved.

4.1 Binary Cross-Entropy#

Used when there are exactly two classes (e.g., 0 or 1, spam or not spam). Also called log loss.

BCE = -(1/N) * sum( y_true_i * log(y_pred_i) + (1 - y_true_i) * log(1 - y_pred_i) )  for i in 1..N

Where:

• y_true_i is the actual label (0 or 1)
• y_pred_i is the predicted probability of the positive class (between 0 and 1), typically produced by a sigmoid activation in the output layer

How it works intuitively:

• If y_true = 1: only the first term y_true * log(y_pred) matters, so the loss is -log(y_pred). This is small when y_pred is close to 1, and large when y_pred is close to 0.
• If y_true = 0: only the second term matters, so the loss is -log(1 - y_pred). This is small when y_pred is close to 0, and large when y_pred is close to 1.

Without binary cross-entropy (using MSE for a binary classification problem): the loss landscape becomes less suited for probability outputs, gradients can be weaker especially when combined with a sigmoid output (a known issue called saturation, where the sigmoid's gradient becomes very small at the extremes, slowing learning).

With binary cross-entropy paired with a sigmoid output: the combination produces a well-behaved gradient that scales naturally with how wrong the prediction is, leading to faster, more stable training for binary classification.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def binary_cross_entropy(y_true, y_pred, epsilon=1e-15):
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)  # avoid log(0)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

y_true = np.array([1, 0, 1, 1])
y_pred = np.array([0.9, 0.1, 0.8, 0.4])

print("Binary Cross-Entropy:", binary_cross_entropy(y_true, y_pred))

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using PyTorch
import torch
import torch.nn as nn

# BCELoss expects probabilities (after sigmoid)
loss_fn = nn.BCELoss()
y_pred = torch.tensor([0.9, 0.1, 0.8, 0.4])
y_true = torch.tensor([1.0, 0.0, 1.0, 1.0])
print("BCE (PyTorch, expects probabilities):", loss_fn(y_pred, y_true).item())

# BCEWithLogitsLoss combines sigmoid + BCE in one numerically stable step
# It expects raw logits, NOT probabilities
logits = torch.tensor([2.2, -2.2, 1.4, -0.4])
loss_fn_logits = nn.BCEWithLogitsLoss()
print("BCEWithLogits (PyTorch, expects raw logits):", loss_fn_logits(logits, y_true).item())

Important practical note: frameworks generally recommend using the "with logits" version (e.g., BCEWithLogitsLoss in PyTorch, or from_logits=True in TensorFlow/Keras) rather than manually applying sigmoid and then computing BCE separately, because combining these two steps internally is more numerically stable.

4.2 Categorical Cross-Entropy#

Used for multi-class classification (more than 2 classes) when labels are one-hot encoded (i.e., the true label is represented as a vector like [0, 0, 1, 0] for class index 2 out of 4 classes).

CCE = -(1/N) * sum( sum(y_true_ic * log(y_pred_ic) for c in classes) for i in 1..N )

Where:

• y_true_ic is 1 if sample i belongs to class c, otherwise 0 (one-hot encoded)
• y_pred_ic is the predicted probability that sample i belongs to class c, typically produced by a softmax activation in the output layer

Since y_true_ic is 0 for all classes except the correct one, this simplifies to just: -log(y_pred) for the predicted probability assigned to the true class.

Without one-hot encoding the labels (feeding raw integer class labels into categorical cross-entropy): the formula's structure requires a full probability vector to compare against, so a raw integer label would not align dimensionally with the predicted probability vector, causing incorrect computation or errors.

With one-hot encoded labels: the true label is expressed as a full probability distribution (100% probability on the correct class, 0% on all others), matching the predicted probability distribution's shape and allowing the cross-entropy formula to be applied correctly across all classes.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def categorical_cross_entropy(y_true, y_pred, epsilon=1e-15):
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(np.sum(y_true * np.log(y_pred), axis=1))

# 3 samples, 4 classes, one-hot encoded true labels
y_true = np.array([
    [0, 0, 1, 0],
    [1, 0, 0, 0],
    [0, 1, 0, 0]
])

y_pred = np.array([
    [0.1, 0.1, 0.7, 0.1],
    [0.6, 0.2, 0.1, 0.1],
    [0.2, 0.5, 0.2, 0.1]
])

print("Categorical Cross-Entropy:", categorical_cross_entropy(y_true, y_pred))

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using TensorFlow/Keras
import tensorflow as tf

loss_fn = tf.keras.losses.CategoricalCrossentropy()
y_true = tf.constant([[0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 0]], dtype=tf.float32)
y_pred = tf.constant([[0.1, 0.1, 0.7, 0.1], [0.6, 0.2, 0.1, 0.1], [0.2, 0.5, 0.2, 0.1]], dtype=tf.float32)
print("Categorical Cross-Entropy (Keras):", loss_fn(y_true, y_pred).numpy())

4.3 Sparse Categorical Cross-Entropy#

Mathematically identical to categorical cross-entropy, but expects the true labels as plain integer class indices (e.g., 2 instead of [0, 0, 1, 0]) rather than one-hot encoded vectors.

Sparse_CCE = -(1/N) * sum( log(y_pred_i, true_class_index_i) for i in 1..N )

This directly looks up the predicted probability for the correct class index, without needing the full one-hot vector.

Without sparse categorical cross-entropy (forced to always one-hot encode labels for categorical cross-entropy): for problems with a large number of classes (e.g., thousands of vocabulary words in NLP, or thousands of image categories), one-hot encoding every label wastes significant memory, since each one-hot vector is mostly zeros.

With sparse categorical cross-entropy: labels can be stored and passed as simple integers, which is far more memory-efficient and convenient, especially for large-class-count problems, while producing mathematically identical results to categorical cross-entropy with one-hot labels.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def sparse_categorical_cross_entropy(y_true, y_pred, epsilon=1e-15):
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    N = len(y_true)
    correct_class_probs = y_pred[np.arange(N), y_true]
    return -np.mean(np.log(correct_class_probs))

# 3 samples, 4 classes, true labels as plain integers (not one-hot)
y_true = np.array([2, 0, 1])  # same true classes as the one-hot example above

y_pred = np.array([
    [0.1, 0.1, 0.7, 0.1],
    [0.6, 0.2, 0.1, 0.1],
    [0.2, 0.5, 0.2, 0.1]
])

print("Sparse Categorical Cross-Entropy:", sparse_categorical_cross_entropy(y_true, y_pred))

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using PyTorch
import torch
import torch.nn as nn

# PyTorch's CrossEntropyLoss is actually "sparse" by default:
# it expects raw logits and integer class labels (not one-hot, not probabilities)
loss_fn = nn.CrossEntropyLoss()

logits = torch.tensor([[0.5, 0.3, 2.1, 0.2],
                        [1.8, 0.4, 0.1, 0.3],
                        [0.3, 1.5, 0.4, 0.2]])
y_true = torch.tensor([2, 0, 1])  # integer class indices

print("CrossEntropyLoss (PyTorch, integer labels + raw logits):", loss_fn(logits, y_true).item())

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Using TensorFlow/Keras
import tensorflow as tf

loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()
y_true = tf.constant([2, 0, 1])
y_pred = tf.constant([[0.1, 0.1, 0.7, 0.1], [0.6, 0.2, 0.1, 0.1], [0.2, 0.5, 0.2, 0.1]], dtype=tf.float32)
print("Sparse Categorical Cross-Entropy (Keras):", loss_fn(y_true, y_pred).numpy())

Important framework-specific note: PyTorch's nn.CrossEntropyLoss is effectively the "sparse" version by default, since it always expects integer class labels combined with raw logits (it applies log-softmax internally). PyTorch does not have a separate "CategoricalCrossEntropyLoss" that takes one-hot labels; if you have one-hot encoded labels in PyTorch, you would need to convert them to integer indices first (e.g., using torch.argmax) before using CrossEntropyLoss.

Classification Losses Comparison#

---

Quick Recap (Beginner to Advanced Flow)#

1. A loss function measures the error between a single prediction and its true value.
2. The objective function is the broader quantity being minimized during training, typically the average loss across the dataset, optionally combined with regularization terms.
3. Regression losses (MSE, MAE, Huber) are used for continuous targets:
   • MSE penalizes large errors heavily but is sensitive to outliers.
   • MAE treats all errors linearly and is more robust to outliers.
   • Huber Loss blends both, behaving like MSE for small errors and MAE for large errors.
4. Classification losses are built on the concept of cross-entropy, which measures the difference between the true and predicted probability distributions:
   • Binary Cross-Entropy is used for two-class problems with sigmoid outputs.
   • Categorical Cross-Entropy is used for multi-class problems with one-hot encoded labels and softmax outputs.
   • Sparse Categorical Cross-Entropy is mathematically identical to categorical cross-entropy but uses integer class labels instead of one-hot vectors, saving memory for problems with many classes.