Gradient Descent: Complete Notes (Beginner to Advanced)

---

1. Gradient Descent#

Gradient descent is the core optimization algorithm used to train neural networks. Its job is to iteratively adjust the network's weights and biases to minimize the loss function, using the gradients computed by backpropagation.

The core update rule:

$\theta_{t+1} = \theta_t - \eta \cdot \nabla_\theta \mathcal{L}(\theta_t)$

Where:

• theta represents any parameter (a weight or bias)
• dL/dtheta is the gradient of the loss with respect to that parameter (computed via backpropagation)
• learning_rate is a small positive number controlling how big each update step is

Intuition: imagine standing on a hilly landscape (the loss surface) in thick fog, unable to see anything except the ground right under your feet. The gradient tells you the direction of steepest ascent from where you're standing. To reach the lowest point (minimum loss), you take a step in the opposite direction of the gradient, then repeat this process from your new position, over and over.

Why we subtract the gradient (not add it): the gradient points toward the direction of steepest increase in the loss. Since the goal is to minimize the loss, moving in the negative gradient direction is what reduces the loss.

Without gradient descent: you would have gradients from backpropagation (telling you the direction of the loss's steepest increase for every parameter) but no defined mechanism to actually use that information to update the parameters and improve the model over time.

With gradient descent: you have a concrete, repeatable procedure that uses the gradient to iteratively move the parameters toward values that reduce the loss, which is exactly how a network "learns" from data.

The general training loop:

code
Ask AICopy
repeat until convergence (or a fixed number of epochs):
    1. Forward pass: compute predictions and loss
    2. Backward pass: compute gradients via backpropagation
    3. Update: adjust each parameter using the gradient descent rule

Code example (gradient descent on a simple function, no neural network yet):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

# Minimize f(x) = x^2, whose derivative is f'(x) = 2x
def f(x):
    return x ** 2

def f_derivative(x):
    return 2 * x

x = 10.0                 # starting point, far from the minimum (x=0)
learning_rate = 0.1
steps = 30

for step in range(steps):
    grad = f_derivative(x)
    x = x - learning_rate * grad
    if step % 5 == 0:
        print(f"Step {step}: x = {x:.5f}, f(x) = {f(x):.5f}")

print("Final x (should be close to 0):", x)

---

2. Batch Gradient Descent#

Batch gradient descent (sometimes just called "Gradient Descent" or "Full-Batch Gradient Descent") computes the gradient using the entire training dataset at once, before making a single parameter update.

code
Ask AICopy
For one update step:
    gradient = (1/N) * sum(gradient_of_loss_for_sample_i for i in 1..N)
    theta_new = theta_old - learning_rate * gradient

Where N is the total number of training samples.

Characteristics:

• Every single update step uses information from the entire dataset, so the gradient direction is the most accurate estimate of the true gradient possible.
• The loss decreases smoothly and consistently step by step, without much noise or fluctuation.
• Only one weight update happens per epoch (one full pass through the dataset), since the entire dataset must be processed before a single update can occur.

Without batch gradient descent (skipping the idea of using the full dataset for a stable gradient estimate): you would have no baseline for what an "exact" gradient computation even looks like, since it's the most straightforward, mathematically precise way to compute the true gradient of the loss function over the whole dataset.

With batch gradient descent: you get the most stable and accurate gradient direction possible at each step, but at a steep computational cost, since you must process every single training sample before making even one update. For large datasets (millions of samples), this makes training extremely slow, and it also requires the entire dataset to fit in memory at once.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

# Batch Gradient Descent for simple linear regression: y = w*x + b
np.random.seed(0)
X = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y_true = np.array([3.0, 5.0, 7.0, 9.0, 11.0])  # true relationship: y = 2x + 1

w, b = 0.0, 0.0
learning_rate = 0.01
epochs = 100
N = len(X)

for epoch in range(epochs):
    y_pred = w * X + b
    error = y_pred - y_true

    # Gradient uses ALL samples at once (full batch)
    dw = (2 / N) * np.sum(error * X)
    db = (2 / N) * np.sum(error)

    w -= learning_rate * dw
    b -= learning_rate * db

print(f"Learned: w = {w:.3f}, b = {b:.3f} (target: w=2, b=1)")

---

3. Stochastic Gradient Descent#

Stochastic Gradient Descent (SGD) is the opposite extreme from batch gradient descent: it computes the gradient and updates the parameters using just one randomly chosen training sample at a time.

Mathematical Formulation
Copy
For each individual sample i (processed one at a time, ideally in random order):
    gradient = gradient_of_loss_for_sample_i
    theta_new = theta_old - learning_rate * gradient

This means that for a dataset with N samples, one epoch results in N separate weight updates, one after every single sample.

Characteristics:

• Much faster per-update, since each update only requires processing one sample.
• The path toward the minimum is noisy and fluctuates, since each individual sample's gradient is a rough, noisy estimate of the true gradient over the whole dataset (a single sample may not be representative of the overall pattern).
• This noise is not purely a downside: it can actually help the optimizer escape shallow local minima or saddle points that batch gradient descent might get stuck in.

Without SGD (relying only on batch gradient descent): for very large datasets, you would need to wait for a full pass through millions of samples just to make a single update, which is prohibitively slow and does not scale to modern large datasets.

With SGD: the model starts learning and improving from the very first sample, updates happen extremely frequently, and the overall training can converge faster in wall-clock time, especially for very large datasets, at the cost of a noisier, less stable convergence path.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

np.random.seed(0)
X = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y_true = np.array([3.0, 5.0, 7.0, 9.0, 11.0])

w, b = 0.0, 0.0
learning_rate = 0.01
epochs = 100
N = len(X)

for epoch in range(epochs):
    indices = np.random.permutation(N)  # shuffle order each epoch
    for i in indices:
        x_i = X[i]
        y_i = y_true[i]

        y_pred = w * x_i + b
        error = y_pred - y_i

        # Gradient uses only ONE sample
        dw = 2 * error * x_i
        db = 2 * error

        w -= learning_rate * dw
        b -= learning_rate * db

print(f"Learned: w = {w:.3f}, b = {b:.3f} (target: w=2, b=1)")

Important terminology note: in modern deep learning frameworks (PyTorch's optim.SGD, for example), "SGD" is often used loosely to refer to the general gradient descent update rule (theta = theta - lr * gradient) regardless of whether it's applied per-sample or per-mini-batch. In practice, true one-sample-at-a-time SGD is rarely used directly; mini-batch gradient descent (below) is what's actually used in nearly all real-world training, but it's frequently still just called "SGD" in framework APIs and papers.

---

4. Mini-Batch Gradient Descent#

Mini-batch gradient descent is the practical middle ground between batch gradient descent (entire dataset per update) and stochastic gradient descent (one sample per update). It computes the gradient using a small, randomly sampled subset (batch) of the training data at each update step.

code
Ask AICopy
For each mini-batch of size B (a small subset of the N total samples):
    gradient = (1/B) * sum(gradient_of_loss_for_sample_i for i in the batch)
    theta_new = theta_old - learning_rate * gradient

Common batch sizes are powers of 2 for computational efficiency on GPUs: 32, 64, 128, 256, etc.

Characteristics:

• Number of updates per epoch = N / B (total samples divided by batch size).
• Strikes a balance: the gradient estimate is more stable than single-sample SGD (since it's averaged over multiple samples) but still much faster to compute than a full batch gradient descent update.
• Enables efficient use of parallel hardware (GPUs), since matrix operations on a batch of samples can be computed simultaneously rather than one at a time.
• This is the standard approach used in virtually all real-world deep learning training, including for training large models like transformers.

Without mini-batch gradient descent: you'd be forced to choose between batch gradient descent's stability-but-impractical-slowness on large datasets, or pure SGD's speed-but-high-noise/inefficient-hardware-usage. Neither extreme is well-suited to how modern hardware (GPUs/TPUs) is designed to process data in parallel.

With mini-batch gradient descent: you get a practical, tunable trade-off between gradient stability and computational speed, while also taking full advantage of parallelized matrix operations on GPUs, which is why it's the default choice for training essentially all modern neural networks.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

np.random.seed(0)
X = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0])
y_true = 2 * X + 1  # true relationship

w, b = 0.0, 0.0
learning_rate = 0.01
epochs = 100
batch_size = 3
N = len(X)

for epoch in range(epochs):
    indices = np.random.permutation(N)
    X_shuffled = X[indices]
    y_shuffled = y_true[indices]

    for start in range(0, N, batch_size):
        end = start + batch_size
        X_batch = X_shuffled[start:end]
        y_batch = y_shuffled[start:end]
        B = len(X_batch)

        y_pred = w * X_batch + b
        error = y_pred - y_batch

        # Gradient averaged over the mini-batch
        dw = (2 / B) * np.sum(error * X_batch)
        db = (2 / B) * np.sum(error)

        w -= learning_rate * dw
        b -= learning_rate * db

print(f"Learned: w = {w:.3f}, b = {b:.3f} (target: w=2, b=1)")

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Mini-batch gradient descent using PyTorch's DataLoader (standard real-world approach)
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

X = torch.arange(1.0, 9.0).unsqueeze(1)   # shape (8, 1)
y_true = 2 * X + 1

dataset = TensorDataset(X, y_true)
dataloader = DataLoader(dataset, batch_size=3, shuffle=True)  # mini-batches of size 3

model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(100):
    for X_batch, y_batch in dataloader:      # loops through mini-batches
        y_pred = model(X_batch)
        loss = loss_fn(y_pred, y_batch)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

print("Learned weight:", model.weight.item())
print("Learned bias:", model.bias.item())

Comparison: Batch vs Stochastic vs Mini-Batch#

---

5. Learning Rate#

The learning rate is a hyperparameter that controls how large each parameter update step is during gradient descent. It directly scales the gradient before it's subtracted from the current parameter value.

theta_new = theta_old - learning_rate * gradient

Effect of learning rate size:

• Too small a learning rate: each update step is tiny, so the model converges extremely slowly, potentially requiring an impractically large number of epochs to reach a good solution. Training could also get stuck in a poor local minimum simply because it doesn't have enough "energy" to move past it.
• Too large a learning rate: each update step overshoots the minimum, potentially causing the loss to oscillate wildly or even increase and diverge entirely (in severe cases, causing the loss to become NaN).
• A well-tuned learning rate: allows the model to converge steadily and efficiently to a good minimum within a reasonable number of epochs.

Without carefully choosing a learning rate: the model may either take an impractically long time to train (if too small) or fail to train at all, with the loss diverging instead of decreasing (if too large). The learning rate is widely regarded as one of the single most important hyperparameters to tune in deep learning.

With a well-chosen learning rate (often found through experimentation, a learning rate range test, or automated hyperparameter search): training converges efficiently to a good solution within a practical amount of time.

Visual intuition (loss vs. learning rate size):

Mathematical Formulation
Copy
Too small LR:    loss ------\
                              \____________________ (very slow decrease)

Good LR:         loss ----\
                            \
                             \______ (steady, efficient decrease)

Too large LR:    loss --\  /\    /\
                          \/  \  /  \    (oscillating or diverging)
                              \/    \

Code example (comparing different learning rates on the same problem):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def f(x):
    return x ** 2

def f_derivative(x):
    return 2 * x

def run_gradient_descent(learning_rate, steps=20, start=10.0):
    x = start
    history = [x]
    for _ in range(steps):
        x = x - learning_rate * f_derivative(x)
        history.append(x)
    return history

print("Too small LR (0.001):", run_gradient_descent(0.001)[-1])
print("Good LR (0.1):", run_gradient_descent(0.1)[-1])
print("Too large LR (1.1):", run_gradient_descent(1.1)[-1])  # will diverge/oscillate

Running this, you'll notice the learning rate of 0.001 barely moves x from 10 after 20 steps, the learning rate of 0.1 converges smoothly close to 0, and the learning rate of 1.1 causes x to oscillate and grow in magnitude rather than converge (since 1.1 > 1 causes each step to overshoot further than the previous position, for this particular function).

Setting the learning rate in practice:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

model = nn.Linear(10, 1)

# The learning rate (lr) is set directly on the optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
# or, a common default for the Adam optimizer:
optimizer_adam = torch.optim.Adam(model.parameters(), lr=0.001)

---

6. Learning Rate Decay#

Learning rate decay (also called learning rate scheduling) is the strategy of reducing the learning rate over the course of training, rather than keeping it fixed at a single value throughout.

The underlying intuition: early in training, the model's weights are far from optimal, so large update steps (a higher learning rate) help it make fast progress toward a good region of the loss landscape. Later in training, as the model gets close to a good minimum, smaller, more careful update steps (a lower learning rate) help it settle precisely into that minimum without overshooting or oscillating around it.

Without learning rate decay (using a single fixed learning rate for the entire training run): you are forced into a compromise. A learning rate large enough for fast initial progress may be too large to let the model settle precisely near the end of training, causing it to bounce around the minimum instead of converging tightly. Conversely, a learning rate small enough for fine, precise convergence near the end would make the early stages of training unnecessarily slow.

With learning rate decay: you get the best of both scenarios: fast progress early in training when a large learning rate is beneficial, and fine, stable convergence later in training when a small learning rate is beneficial, all within the same training run.

Common learning rate decay strategies:

6.1 Step Decay#

Reduce the learning rate by a fixed factor after a set number of epochs.

learning_rate = initial_lr * decay_factor^floor(epoch / decay_step)

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn
from torch.optim.lr_scheduler import StepLR

model = nn.Linear(10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# Reduce learning rate by a factor of 0.5 every 10 epochs
scheduler = StepLR(optimizer, step_size=10, gamma=0.5)

for epoch in range(30):
    # ... training steps happen here (forward, loss, backward, optimizer.step()) ...
    scheduler.step()  # update the learning rate according to the schedule
    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Learning Rate: {optimizer.param_groups[0]['lr']}")

6.2 Exponential Decay#

Continuously reduce the learning rate by a fixed multiplicative factor every epoch (a smoother version of step decay).

learning_rate = initial_lr * decay_rate^epoch

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
from torch.optim.lr_scheduler import ExponentialLR

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = ExponentialLR(optimizer, gamma=0.95)  # multiply LR by 0.95 every epoch

for epoch in range(10):
    scheduler.step()
    print(f"Epoch {epoch}, Learning Rate: {optimizer.param_groups[0]['lr']:.5f}")

6.3 Cosine Annealing#

Smoothly decreases the learning rate following a cosine curve, from the initial value down to near zero (or a specified minimum) over a set number of epochs. This has become a very popular choice in modern deep learning, especially for training large models.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
from torch.optim.lr_scheduler import CosineAnnealingLR

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = CosineAnnealingLR(optimizer, T_max=30)  # decays over 30 epochs

for epoch in range(30):
    scheduler.step()

6.4 Reduce on Plateau#

Rather than following a fixed schedule, this strategy monitors a metric (typically the validation loss) and reduces the learning rate only when that metric stops improving for a specified number of epochs (called "patience"). This adapts the schedule to the model's actual training progress rather than a predetermined timeline.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
from torch.optim.lr_scheduler import ReduceLROnPlateau

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=5)

for epoch in range(50):
    # ... training happens, and you compute validation_loss ...
    validation_loss = 0.5  # placeholder value
    scheduler.step(validation_loss)  # reduces LR if validation_loss hasn't improved for 5 epochs

Learning Rate Decay Strategies Comparison#

---

Quick Recap (Beginner to Advanced Flow)#

1. Gradient descent is the core algorithm that updates parameters by moving them in the opposite direction of the loss gradient, scaled by the learning rate.
2. Batch gradient descent uses the entire dataset per update: accurate but slow and memory-intensive.
3. Stochastic gradient descent (SGD) uses one sample per update: fast but noisy.
4. Mini-batch gradient descent uses a small subset per update, balancing accuracy and speed, and is the standard approach in nearly all real-world deep learning.
5. The learning rate controls the size of each update step; too small slows training drastically, too large causes divergence or oscillation.
6. Learning rate decay reduces the learning rate over the course of training, allowing fast early progress and precise, stable convergence later on, through strategies like step decay, exponential decay, cosine annealing, or adaptive reduce-on-plateau schedules.