Model Training & Evaluation: Complete Notes (Beginner to Advanced)

---

Introduction#

Model training is the process of learning model parameters from training data, while model evaluation is the process of measuring how well the trained model performs on data it did not use to update its parameters.

A typical machine-learning workflow is:

text
Ask AICopy
Raw Data
   ↓
Data Preparation
   ↓
Training / Validation / Test Split
   ↓
Training
   ↓
Validation
   ↓
Hyperparameter Tuning
   ↓
Final Model
   ↓
Test Evaluation

The key principle is:

text
Ask AICopy
Training Set   → Learn parameters
Validation Set → Choose/tune the model
Test Set       → Final unbiased evaluation

---

1. Training Set

1.1 What is a Training Set?#

The training set is the portion of the dataset used to learn the model's parameters.

For a neural network, these parameters include:

›Weights
Biases

During training:

text
Ask AICopy
Input
  ↓
Model
  ↓
Prediction
  ↓
Loss
  ↓
Gradients
  ↓
Parameter Update

The model repeatedly sees training examples and adjusts its parameters to reduce the training loss.

---

1.2 Example#

Suppose a dataset contains:

›10,000 samples

A possible split is:

text
Ask AICopy
Training → 8,000
Validation → 1,000
Test → 1,000

The exact proportions are not fixed and depend on the dataset and task.

---

2. Validation Set

2.1 What is a Validation Set?#

The validation set is data used during model development to evaluate choices such as:

• Hyperparameters
• Architecture
• Regularization
• Training duration
• Decision thresholds

The validation set should not be used to directly update model parameters.

Conceptually:

text
Ask AICopy
Training Data
     ↓
Learn weights

Validation Data
     ↓
Choose the better configuration

---

2.2 Why Do We Need Validation Data?#

Suppose you train several models:

Mathematical Formulation
Copy
Model A → validation accuracy = 90%
Model B → validation accuracy = 94%
Model C → validation accuracy = 91%

You can select Model B based on validation performance.

However, repeatedly making decisions based on the same validation set means the validation set is no longer a completely untouched source of evidence.

This is why a separate test set is useful for final evaluation.

---

3. Test Set

3.1 What is a Test Set?#

The test set is reserved for the final evaluation of the selected model.

The test set should not be used for:

• Learning model parameters
• Choosing hyperparameters
• Selecting the best architecture
• Repeated model development decisions

The basic idea is:

text
Ask AICopy
Training → Learn
Validation → Select
Test → Final evaluation

---

3.2 Why Keep the Test Set Separate?#

Suppose you repeatedly evaluate models on the test set and choose the model with the best test score.

The test set has effectively become part of the model-selection process.

Therefore, the reported test performance may become overly optimistic.

A properly held-out test set provides a better estimate of generalization to unseen data.

---

4. Training Loop

4.1 What is a Training Loop?#

A training loop is the repeated process through which a model learns from batches of training data.

The basic loop is:

text
Ask AICopy
Get Batch
   ↓
Forward Pass
   ↓
Prediction
   ↓
Calculate Loss
   ↓
Backpropagation
   ↓
Calculate Gradients
   ↓
Optimizer Update
   ↓
Next Batch

---

4.2 Basic PyTorch Training Loop#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
for X_batch, y_batch in train_loader:

    optimizer.zero_grad()

    predictions = model(X_batch)

    loss = loss_fn(
        predictions,
        y_batch
    )

    loss.backward()

    optimizer.step()

The important operations are:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
optimizer.zero_grad()

Clears previously accumulated gradients.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
predictions = model(X_batch)

Performs the forward pass.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
loss = loss_fn(predictions, y_batch)

Measures prediction error.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
loss.backward()

Computes gradients.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
optimizer.step()

Updates trainable parameters.

---

5. Validation Loop

5.1 What is a Validation Loop?#

A validation loop evaluates the current model using validation data without updating its parameters.

Conceptually:

text
Ask AICopy
Validation Batch
      ↓
Forward Pass
      ↓
Prediction
      ↓
Loss / Metrics
      ↓
No Parameter Update

---

5.2 PyTorch Validation Loop#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
model.eval()

with torch.no_grad():

    for X_batch, y_batch in val_loader:

        predictions = model(X_batch)

        loss = loss_fn(
            predictions,
            y_batch
        )

model.eval() changes layers such as dropout and batch normalization to evaluation behavior.

torch.no_grad() prevents unnecessary gradient tracking.

---

5.3 Training vs Validation#

---

6. Epochs

6.1 What is an Epoch?#

An epoch is one complete pass through the training dataset.

Suppose:

Mathematical Formulation
Copy
Training samples = 1,000
Batch size = 100

Then approximately:

Mathematical Formulation
Copy
10 batches = 1 epoch

If training runs for:

›20 epochs

the model processes the training dataset approximately 20 times.

---

6.2 Epoch vs Iteration#

An iteration usually refers to one optimizer update associated with one batch.

Example:

Mathematical Formulation
Copy
1 epoch
=
10 batches
=
approximately 10 training iterations

assuming every batch produces one optimizer update.

---

7. Batch Size

7.1 What is Batch Size?#

Batch size is the number of training samples processed in one training step.

Example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
batch_size = 32

means each batch contains up to 32 samples.

---

7.2 Small vs Large Batch#

Small Batch#

Advantages:

• Lower memory usage
• More frequent parameter updates
• Can introduce useful gradient noise

Disadvantages:

• More iterations per epoch
• Training may be less computationally efficient on some hardware

Large Batch#

Advantages:

• Better hardware utilization in suitable workloads
• Fewer optimizer updates per epoch
• Can improve throughput

Disadvantages:

• Higher memory usage
• May require learning-rate adjustment
• Can sometimes affect generalization

There is no universally best batch size.

---

8. Hyperparameters

8.1 What are Hyperparameters?#

Hyperparameters are settings chosen by the practitioner rather than learned directly as ordinary model parameters during backpropagation.

Examples include:

text
Ask AICopy
Learning rate
Batch size
Number of epochs
Optimizer
Weight decay
Dropout rate
Model depth
Model width
LoRA rank

---

8.2 Parameters vs Hyperparameters#

Parameters#

Learned from data:

›Weights
Biases

Hyperparameters#

Chosen before or during the training process:

text
Ask AICopy
Learning rate
Batch size
Number of layers
Dropout rate

Conceptually:

text
Ask AICopy
Hyperparameters
      ↓
Training Process
      ↓
Learned Parameters
      ↓
Model

---

9. Hyperparameter Tuning

9.1 What is Hyperparameter Tuning?#

Hyperparameter tuning is the process of testing different hyperparameter configurations and selecting one that performs well on validation data.

Example:

text
Ask AICopy
Learning Rate
0.1
0.01
0.001
0.0001

Train/evaluate each configuration:

text
Ask AICopy
0.1    → validation score 82%
0.01   → validation score 91%
0.001  → validation score 94%
0.0001 → validation score 88%

Choose:

›0.001

based on the validation result.

---

9.2 Grid Search#

Grid search evaluates combinations from predefined sets.

Example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
learning_rates = [0.01, 0.001]
batch_sizes = [32, 64]

Possible combinations:

text
Ask AICopy
0.01  + 32
0.01  + 64
0.001 + 32
0.001 + 64

---

9.3 Random Search#

Random search samples configurations from specified distributions or ranges.

It can explore large hyperparameter spaces more efficiently than exhaustive grid search in many situations.

---

9.4 Bayesian Optimization#

Bayesian optimization uses previous evaluation results to decide which hyperparameter configuration to try next.

Conceptually:

text
Ask AICopy
Try Configuration
       ↓
Observe Validation Score
       ↓
Update Search Model
       ↓
Choose Promising Configuration
       ↓
Try Again

---

9.5 Hyperparameter Tuning Workflow#

text
Ask AICopy
Choose Search Space
        ↓
Generate Configurations
        ↓
Train Models
        ↓
Evaluate on Validation Set
        ↓
Select Best Configuration
        ↓
Train Final Model
        ↓
Evaluate Once on Test Set

---

10. Cross-Validation

10.1 What is Cross-Validation?#

Cross-validation evaluates a model across multiple train/validation splits rather than relying on one fixed validation split.

The most common form is K-Fold Cross-Validation.

---

10.2 K-Fold Cross-Validation#

Suppose:

Mathematical Formulation
Copy
K = 5

The dataset is divided into five folds:

text
Ask AICopy
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

Training and validation are performed five times.

text
Ask AICopy
Run 1:
Validation → Fold 1
Training   → Folds 2,3,4,5

Run 2:
Validation → Fold 2
Training   → Folds 1,3,4,5

Run 3:
Validation → Fold 3
Training   → Folds 1,2,4,5

Run 4:
Validation → Fold 4
Training   → Folds 1,2,3,5

Run 5:
Validation → Fold 5
Training   → Folds 1,2,3,4

Then average the evaluation scores.

Mathematical Formulation
Copy
CV Score
=
(score1 + score2 + ... + scoreK) / K

---

10.3 Why Use Cross-Validation?#

It can provide a more stable estimate of model performance, especially when the dataset is relatively small.

Instead of depending heavily on one arbitrary train/validation split:

›One split

we use:

›Multiple splits

---

10.4 Stratified Cross-Validation#

For classification, Stratified K-Fold attempts to preserve class proportions across folds.

For example:

text
Ask AICopy
Dataset:
90% Class A
10% Class B

Each fold attempts to maintain approximately the same class distribution.

This is especially useful for imbalanced classification datasets.

---

10.5 Cross-Validation in Deep Learning#

K-fold cross-validation can be computationally expensive for large neural networks because the model must be trained multiple times.

Therefore, a fixed train/validation split is often more practical for large-scale deep-learning training.

---

11. Learning Curves

11.1 What are Learning Curves?#

Learning curves show how model performance changes as the amount of training data increases.

Typical structure:

text
Ask AICopy
Training Set Size
       ↓
Train Model
       ↓
Measure Training Performance
       ↓
Measure Validation Performance
       ↓
Plot Scores

Example:

›Training samples:
1k → 2k → 4k → 8k → 16k

For each size, record:

›Training score
Validation score

---

11.2 What Can Learning Curves Tell Us?#

Learning curves can help identify:

• High bias
• High variance
• Whether more training data may help
• Whether the model has enough capacity

A large gap between training and validation performance can indicate overfitting.

If both training and validation performance are poor, the model may have high bias or insufficient capacity.

---

12. Loss Curves

12.1 What is a Loss Curve?#

A loss curve shows how loss changes during training.

Typically:

›X-axis → Epoch
Y-axis → Loss

You may plot:

›Training Loss
Validation Loss

Example:

Mathematical Formulation
Copy
Loss
 │\
 │ \
 │  \        Training
 │   \______
 │
 │   \____   Validation
 │       \__
 └────────────── Epoch

---

12.2 Interpreting Loss Curves#

Healthy Training#

›Training Loss ↓
Validation Loss ↓

Both improve.

Overfitting#

text
Ask AICopy
Training Loss ↓ continuously

Validation Loss ↓
                 ↑
                 └── starts increasing

The model continues fitting training data while validation performance deteriorates.

Underfitting#

›Training Loss → remains high
Validation Loss → remains high

The model is not learning the underlying pattern sufficiently.

---

12.3 Early Stopping#

Loss curves can be used to determine when to stop training.

For example:

Mathematical Formulation
Copy
Epoch 1 → val_loss = 0.50
Epoch 2 → val_loss = 0.42
Epoch 3 → val_loss = 0.35
Epoch 4 → val_loss = 0.31
Epoch 5 → val_loss = 0.34

The validation loss is best at epoch 4.

A training process can use early stopping to stop after validation performance stops improving.

---

13. Evaluation Metrics

13.1 What are Evaluation Metrics?#

Metrics quantify model performance.

The correct metric depends on the task.

---

13.2 Classification Metrics#

Accuracy#

Mathematical Formulation
Copy
Accuracy
=
Correct Predictions
-------------------
Total Predictions

Accuracy works well when classes are reasonably balanced.

---

Precision#

Mathematical Formulation
Copy
Precision
=
TP
--------
TP + FP

It answers:

›Of the examples predicted as positive,
how many were actually positive?

---

Recall#

Mathematical Formulation
Copy
Recall
=
TP
--------
TP + FN

It answers:

›Of the actual positive examples,
how many did the model identify?

---

F1 Score#

Mathematical Formulation
Copy
F1
=
2 × Precision × Recall
-----------------------
Precision + Recall

F1 balances precision and recall through their harmonic mean.

---

13.3 Confusion Matrix#

A binary classification confusion matrix contains:

text
Ask AICopy
                 Predicted
              Positive Negative
Actual Positive   TP       FN
       Negative   FP       TN

Where:

Mathematical Formulation
Copy
TP = True Positive
TN = True Negative
FP = False Positive
FN = False Negative

---

13.4 ROC-AUC#

ROC-AUC measures ranking performance across classification thresholds using:

text
Ask AICopy
True Positive Rate
vs
False Positive Rate

AUC is the area under the ROC curve.

For highly imbalanced classification, PR-AUC can sometimes be more informative than ROC-AUC because it focuses directly on precision and recall behavior.

---

13.5 Regression Metrics#

Mean Absolute Error#

Mathematical Formulation
Copy
MAE
=
(1/n) Σ |y - ŷ|

It measures the average absolute prediction error.

Mean Squared Error#

Mathematical Formulation
Copy
MSE
=
(1/n) Σ(y - ŷ)²

Large errors receive greater penalty because the error is squared.

Root Mean Squared Error#

Mathematical Formulation
Copy
RMSE = √MSE

RMSE has the same units as the target variable.

R²#

Mathematical Formulation
Copy
R²
=
1 - SS_res / SS_tot

It measures the proportion of variance explained relative to a baseline based on the target mean.

---

14. Class Imbalance

14.1 What is Class Imbalance?#

Class imbalance occurs when some classes contain substantially more examples than others.

Example:

›Class 0 → 9,500 samples
Class 1 →   500 samples

Distribution:

›Class 0 → 95%
Class 1 → 5%

---

14.2 Why Is Imbalance a Problem?#

Suppose a model predicts:

›Class 0 for every sample

It could achieve:

›95% accuracy

while completely failing to detect Class 1.

Therefore, accuracy alone may be misleading.

---

14.3 Better Metrics#

For imbalanced classification, consider:

text
Ask AICopy
Precision
Recall
F1
PR-AUC
ROC-AUC
Confusion Matrix

The most appropriate metric depends on the business or application objective.

---

14.4 Handling Class Imbalance#

Common approaches include:

Class Weights#

Give greater loss weight to underrepresented classes.

In PyTorch:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
class_weights = torch.tensor(
    [1.0, 5.0]
)

loss_fn = nn.CrossEntropyLoss(
    weight=class_weights
)

The exact weights should be chosen carefully.

---

Oversampling#

Increase the representation of minority examples in training.

Undersampling#

Reduce the number of majority-class examples.

Data Augmentation#

Create additional training examples for the minority class where appropriate.

Threshold Adjustment#

For probabilistic classifiers, adjust the decision threshold to match the desired precision/recall trade-off.

---

14.5 Important Rule#

Do not blindly resample the entire dataset before splitting.

Instead:

text
Ask AICopy
Split data
   ↓
Keep validation/test representative
   ↓
Apply resampling only to training data

Otherwise, evaluation can become misleading.

---

15. Data Leakage

15.1 What is Data Leakage?#

Data leakage occurs when information that should not be available to the model during training becomes available through the training process.

Leakage can produce unrealistically strong validation or test performance.

---

15.2 Example: Scaling Leakage#

Suppose the full dataset is standardized before splitting:

text
Ask AICopy
Entire Dataset
     ↓
Calculate mean/std
     ↓
Scale entire dataset
     ↓
Train/Test Split

The scaling statistics were calculated using information from the test set.

A safer approach is:

text
Ask AICopy
Split Dataset
     ↓
Fit scaler on Training Data
     ↓
Transform Training Data
     ↓
Transform Validation/Test using same training-fitted scaler

Example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)

---

15.3 Example: Duplicate Data#

Suppose nearly identical records appear in both training and test sets.

The model may effectively see the same information during training and testing.

This can cause artificially high test performance.

---

15.4 Example: Future Information#

For time-dependent prediction:

›Predict tomorrow's value

but accidentally include:

›information from tomorrow

as an input feature.

That is leakage because the feature would not actually be available at prediction time.

---

15.5 Leakage Through Validation/Test Data#

Do not repeatedly use test results to make modeling decisions.

Correct:

text
Ask AICopy
Training → Learn
Validation → Tune
Test → Final evaluation

Incorrect:

text
Ask AICopy
Training
 ↓
Test
 ↓
Change model
 ↓
Test again
 ↓
Change model
 ↓
Test again

The test set is gradually influencing model development.

---

16. Experiment Tracking

16.1 What is Experiment Tracking?#

Experiment tracking is the systematic recording of model experiments so that results can be compared, reproduced, and analyzed.

A single experiment may contain:

text
Ask AICopy
Dataset version
Model architecture
Hyperparameters
Training metrics
Validation metrics
Test metrics
Code version
Random seed
Checkpoint
Notes

---

16.2 Why Track Experiments?#

Suppose you run:

Mathematical Formulation
Copy
Experiment 1
Learning rate = 0.01
Accuracy = 88%

Experiment 2
Learning rate = 0.001
Accuracy = 93%

Experiment 3
Learning rate = 0.0001
Accuracy = 90%

Without tracking, it becomes difficult to remember which configuration produced which result.

Experiment tracking provides a history of these runs.

---

16.3 What Should Be Tracked?#

Configuration#

text
Ask AICopy
Learning rate
Batch size
Epochs
Optimizer
Architecture
Dropout
Weight decay

Metrics#

text
Ask AICopy
Training loss
Validation loss
Training accuracy
Validation accuracy
Precision
Recall
F1

Artifacts#

text
Ask AICopy
Model checkpoint
Plots
Configuration files
Predictions
Logs

Reproducibility Information#

text
Ask AICopy
Dataset version
Code version
Random seed
Environment
Library versions

---

16.4 Experiment Tracking Workflow#

text
Ask AICopy
Experiment Configuration
          ↓
        Train
          ↓
     Log Metrics
          ↓
     Save Artifacts
          ↓
       Compare
          ↓
Select Best Experiment

---

16.5 Experiment Tracking Tools#

Common tools include:

text
Ask AICopy
MLflow
Weights & Biases
TensorBoard

These tools can help track metrics, visualize training, compare runs, and manage model artifacts.

---

17. Complete Model Training & Evaluation Workflow

A production-oriented workflow can be represented as:

text
Ask AICopy
                    Raw Dataset
                         │
                         ▼
                Data Validation
                         │
                         ▼
                  Data Splitting
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
         Training    Validation     Test
             │           │           │
             ▼           │           │
        Preprocessing    │           │
             │           │           │
             ▼           │           │
        Model Training   │           │
             │           │           │
             ▼           │           │
        Hyperparameter   │           │
           Tuning ◄──────┘           │
             │                       │
             ▼                       │
        Best Model                   │
             │                       │
             └───────────────────────┤
                                     ▼
                              Final Test
                              Evaluation
                                     │
                                     ▼
                              Report Results
                                     │
                                     ▼
                            Save Model/Metadata

The critical rule is:

›Test data should remain untouched
until the final evaluation.

---

18. Putting Training and Validation Together

A practical deep-learning training process looks like:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
for epoch in range(num_epochs):

    # --------------------
    # Training
    # --------------------
    model.train()

    for X_batch, y_batch in train_loader:

        optimizer.zero_grad()

        predictions = model(X_batch)

        loss = loss_fn(
            predictions,
            y_batch
        )

        loss.backward()

        optimizer.step()


    # --------------------
    # Validation
    # --------------------
    model.eval()

    with torch.no_grad():

        for X_batch, y_batch in val_loader:

            predictions = model(X_batch)

            val_loss = loss_fn(
                predictions,
                y_batch
            )

Then monitor:

text
Ask AICopy
training loss
validation loss
training metrics
validation metrics

and use the results to decide whether to continue training, tune hyperparameters, or select a checkpoint.

---

19. Summary Table

---

20. Quick Recap

text
Ask AICopy
TRAINING
────────
Training Set
    ↓
Batch
    ↓
Forward Pass
    ↓
Loss
    ↓
Backpropagation
    ↓
Optimizer
    ↓
Updated Parameters

text
Ask AICopy
VALIDATION
──────────
Validation Set
    ↓
Forward Pass
    ↓
Metrics / Loss
    ↓
No Parameter Update
    ↓
Tune / Select

text
Ask AICopy
FINAL EVALUATION
────────────────
Selected Model
      ↓
Untouched Test Set
      ↓
Final Metrics
      ↓
Generalization Estimate

text
Ask AICopy
MODEL DEVELOPMENT
──────────────────
Hyperparameters
      ↓
Training
      ↓
Validation
      ↓
Learning Curves
      ↓
Tune
      ↓
Best Configuration
      ↓
Final Test

The most important mental model is:

text
Ask AICopy
Training Set
→ Learn the model

Validation Set
→ Make development decisions

Test Set
→ Measure final performance

Training Loop
→ Update parameters

Validation Loop
→ Measure without updating

Hyperparameters
→ Control training

Cross-Validation
→ Test stability across splits

Learning/Loss Curves
→ Understand training behavior

Metrics
→ Quantify performance

Class Imbalance
→ Prevent misleading evaluation

Data Leakage
→ Prevent invalid evaluation

Experiment Tracking
→ Make experiments reproducible