∇GradientSchool
Intermediate
18 min read
#Transfer Learning#Pretrained Models#Fine-Tuning#Feature Extraction#Layer Freezing#PyTorch

24. Transfer Learning & Domain Adaptation

Leveraging pretrained representations: feature extraction vs full fine-tuning, strategic layer freezing, learning rate warmups, and domain adaptation techniques.

Transfer Learning: Complete Notes (Beginner to Advanced)

1. Transfer Learning#

Transfer learning is a machine learning technique where knowledge learned from one task or dataset is reused for another related task.

Instead of training a neural network entirely from random initialization:

Architecture & Data Flow
Random Initialization
        |
        v
Train on New Dataset
        |
        v
New Model

we start with a model that has already learned useful patterns:

Architecture & Data Flow
Pretrained Model
       |
       v
Adapt to New Task
       |
       v
New Model

Basic Idea#

Suppose a model was trained on a large image dataset.

It may learn:

text
Early layers:
Edges, colors, textures

Middle layers:
Shapes and patterns

Later layers:
Object-specific features

For a new image task, many of the early and middle features may still be useful.

Therefore, instead of learning everything again, we can reuse the pretrained model.

Typical Flow#

Architecture & Data Flow
Large Dataset
     |
     v
Pretrain Model
     |
     v
Pretrained Weights
     |
     v
New Dataset
     |
     v
Adapt / Fine-Tune
     |
     v
Target Model

2. Why Transfer Learning Is Useful#

Training a deep neural network from scratch can require:

  • Large datasets
  • Significant computation
  • Long training times
  • Careful optimization

Transfer learning can reduce these requirements when the source model and target task are sufficiently related.

Without Transfer Learning#

Architecture & Data Flow
Small Dataset
     |
     v
Randomly initialized model
     |
     v
Train everything
     |
     v
Risk of overfitting

With Transfer Learning#

Architecture & Data Flow
Large Dataset
     |
     v
Pretrained Model
     |
     v
Reuse learned features
     |
     v
Small Target Dataset
     |
     v
Adapt model

Main Benefits#

  • Faster convergence
  • Less training data may be required
  • Lower computational cost
  • Often better performance than training from scratch
  • Useful initialization for the target task

Transfer learning is not guaranteed to improve performance. If the source and target domains are poorly related, transferred knowledge can be unhelpful or even harmful.

3. Feature Extraction#

Feature extraction is a transfer-learning approach where the pretrained model is used primarily as a fixed feature extractor.

The pretrained layers are frozen:

Architecture & Data Flow
Pretrained Model
     |
     +---- Frozen layers
     |
     v
Feature Representation
     |
     v
New Task-Specific Head
     |
     v
Prediction

Example#

Suppose a pretrained CNN has:

Architecture & Data Flow
Convolution layers
       |
       v
Feature extractor
       |
       v
Original classifier

For a new classification task, we can:

Architecture & Data Flow
Remove original classifier
        |
        v
Keep pretrained feature extractor
        |
        v
Add new classifier

Example:

text
Image
  |
Pretrained CNN
  |
Frozen Features
  |
New Linear Layer
  |
New Classes

Why Freeze the Backbone?#

The pretrained layers already contain useful representations.

Freezing them:

  • Reduces the number of trainable parameters
  • Reduces computational cost
  • Prevents rapid changes to pretrained features
  • Can reduce overfitting on small datasets

4. Pretrained Models#

A pretrained model is a model whose parameters have already been learned from a previous training process, usually using a large dataset.

Examples in computer vision include:

  • ResNet
  • VGG
  • EfficientNet
  • MobileNet

Examples in NLP include:

  • BERT
  • T5
  • GPT-family models

The pretrained weights provide a useful starting point for a new task.

Example#

A CNN pretrained on a large image dataset might already know useful visual patterns:

Architecture & Data Flow
Pixels
  |
Edges
  |
Textures
  |
Shapes
  |
Object-level features

A new task may only require adapting the later representation to the target labels.

5. Freezing Layers#

Freezing a layer means preventing its parameters from being updated during training.

If a parameter is frozen:

›gradient update -> not applied

Conceptually:

Architecture & Data Flow
Layer 1  -> Frozen
Layer 2  -> Frozen
Layer 3  -> Frozen
Layer 4  -> Trainable
Layer 5  -> Trainable

The model can still perform forward propagation through frozen layers.

They simply do not receive parameter updates.

Why Freeze Layers?#

Common reasons include:

  • Small target dataset
  • Limited compute
  • Useful pretrained representations
  • Preventing overfitting
  • Faster training

6. Freezing Layers in PyTorch#

A common PyTorch pattern is:

🐍 Python
import torch
import torch.nn as nn
from torchvision import models

model = models.resnet18(weights="DEFAULT")

for param in model.parameters():
    param.requires_grad = False

Now the pretrained parameters are frozen.

We can replace the final classification layer:

🐍 Python
num_features = model.fc.in_features

model.fc = nn.Linear(num_features, 10)

The new classification layer is trainable by default.

Conceptually:

›ResNet backbone -> frozen
Final classifier -> trainable

The optimizer should generally receive only the parameters that are intended to be updated:

🐍 Python
optimizer = torch.optim.Adam(
    model.fc.parameters(),
    lr=1e-3
)

7. Unfreezing Layers#

Unfreezing means allowing previously frozen parameters to receive gradient updates.

For example:

Architecture & Data Flow
Initially:

Backbone -> Frozen
Classifier -> Trainable

After initial training:

›Backbone -> Partially Unfrozen
Classifier -> Trainable

This allows the pretrained representation to adapt to the target dataset.

Why Unfreeze Gradually?#

Earlier layers often learn more general patterns, while later layers tend to become more task-specific.

Therefore, a common strategy is:

text
Start:
Freeze most/all backbone

Then:
Unfreeze later layers

If needed:
Unfreeze more layers

This is not a universal rule, but it is a useful practical strategy.

8. Fine-Tuning#

Fine-tuning means continuing training of a pretrained model on a target dataset so that its parameters adapt to the new task.

A typical workflow is:

Architecture & Data Flow
Pretrained Model
      |
      v
Replace / modify task-specific head
      |
      v
Freeze some layers
      |
      v
Train new head
      |
      v
Unfreeze selected layers
      |
      v
Fine-tune with smaller learning rate

Fine-tuning can involve updating:

  • Only the final layers
  • A subset of the backbone
  • Most of the network
  • The entire network

Therefore, feature extraction is a more restrictive form of transfer learning, while fine-tuning allows some pretrained parameters to change.

9. Fine-Tuning Learning Rate#

Fine-tuning commonly uses a smaller learning rate than training a new model from scratch.

Why?

Because the pretrained weights already contain useful information.

A very large learning rate can make the model change those useful representations too aggressively.

Conceptually:

text
Training from scratch:
larger learning rate may be appropriate

Fine-tuning:
smaller learning rate is often preferred

Different parts of the network can also use different learning rates.

For example:

›Pretrained backbone -> 1e-5
New classifier       -> 1e-3

This is called using discriminative learning rates or different parameter-group learning rates.

10. Full Fine-Tuning#

Full fine-tuning means allowing essentially all pretrained model parameters to be updated on the target dataset.

Architecture & Data Flow
Pretrained Model
      |
      v
All layers trainable
      |
      v
Target Dataset
      |
      v
Adapted Model

Example#

Architecture & Data Flow
Layer 1 -> Trainable
Layer 2 -> Trainable
Layer 3 -> Trainable
Layer 4 -> Trainable
Layer 5 -> Trainable
Classifier -> Trainable

The model can adapt its entire representation to the new task.

Advantages#

  • Maximum ability to adapt
  • Useful when source and target domains differ substantially but are still compatible
  • Can achieve strong target-task performance with sufficient data and compute

Disadvantages#

  • More computation
  • More trainable parameters
  • Higher risk of overfitting on small datasets
  • Can overwrite useful pretrained representations
  • Usually requires more careful learning-rate selection

11. Feature Extraction vs Fine-Tuning vs Full Fine-Tuning#

ApproachBackboneNew HeadTrainable parametersTypical use
Feature extractionFrozenTrainableFewSmall dataset / related task
Partial fine-tuningPartially trainableTrainableMediumAdapt later features
Full fine-tuningTrainableTrainableMost/allMore adaptation needed

Visual Comparison#

Architecture & Data Flow
Feature Extraction

[ Frozen Backbone ] -> [ Trainable Head ]


Partial Fine-Tuning

[ Frozen ] -> [ Trainable Backbone Layers ] -> [ Trainable Head ]


Full Fine-Tuning

[ Trainable Backbone ] -> [ Trainable Head ]

12. Domain Adaptation#

Domain adaptation is the process of adapting a model trained on a source domain so that it performs well on a different but related target domain.

A domain can be characterized by its data distribution and task context.

Architecture & Data Flow
Source Domain
     |
     v
Pretrained Model
     |
     v
Adaptation
     |
     v
Target Domain

Example#

Suppose a model is trained using:

›Source:
Professional product photographs

but must work on:

›Target:
Mobile phone photographs taken in real-world conditions

The task may be similar:

›Object classification

but the data distributions differ.

The model may need domain adaptation.

13. Source Domain and Target Domain#

Source Domain#

The domain from which knowledge is transferred.

Architecture & Data Flow
Source data
   |
   v
Source model

Target Domain#

The domain where the model will ultimately be used.

Architecture & Data Flow
Target data
   |
   v
Adapted model

Conceptually:

Architecture & Data Flow
SOURCE DOMAIN
Large / available data
        |
        v
   Pretraining
        |
        v
   Source Model
        |
        v
   Adaptation
        |
        v
TARGET DOMAIN
New distribution
        |
        v
Target Model

14. Domain Shift#

The difference between source and target data distributions is called domain shift.

For example:

text
Source:
Studio images

Target:
Outdoor images

The objects may be the same, but:

text
Lighting
Background
Camera quality
Image style
Object appearance

can differ.

Therefore:

Mathematical Formulation
P_source(x) != P_target(x)

in a simplified view.

Domain adaptation attempts to reduce the performance degradation caused by such differences.

15. Domain Adaptation vs Transfer Learning#

These concepts are closely related but not identical.

Transfer Learning#

A broad concept:

Architecture & Data Flow
Knowledge from source
        |
        v
Reuse in target task/domain

Domain Adaptation#

A more specific setting where the target domain differs from the source domain and adaptation is needed.

Architecture & Data Flow
Source Domain
      |
      v
Model
      |
      v
Different Target Domain
      |
      v
Adaptation

So:

Architecture & Data Flow
Transfer Learning
        |
        +-- Fine-Tuning
        |
        +-- Feature Extraction
        |
        +-- Domain Adaptation
        |
        +-- Other transfer strategies

The exact taxonomy can vary across literature.

16. Simple Transfer Learning Example#

Suppose we have:

text
Pretrained model:
ImageNet classification

Target task:
Classify different types of flowers

The workflow can be:

text
Image
  |
Pretrained CNN
  |
Frozen feature extractor
  |
New flower classifier
  |
Flower class

After training the new classifier, we may unfreeze later CNN layers:

text
Image
  |
Pretrained CNN
  |
Partially fine-tuned features
  |
Flower classifier
  |
Flower class

This allows the model to learn flower-specific visual features.

17. Practical PyTorch Fine-Tuning Example#

🐍 Python
import torch
import torch.nn as nn
from torchvision import models

model = models.resnet18(weights="DEFAULT")

# Freeze pretrained layers
for param in model.parameters():
    param.requires_grad = False

# Replace classifier
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 5)

# Train only classifier initially
optimizer = torch.optim.Adam(
    model.fc.parameters(),
    lr=1e-3
)

After the classifier has learned:

🐍 Python
# Unfreeze the final ResNet block
for param in model.layer4.parameters():
    param.requires_grad = True

optimizer = torch.optim.Adam(
    [
        {"params": model.layer4.parameters(), "lr": 1e-5},
        {"params": model.fc.parameters(), "lr": 1e-3},
    ]
)

This is an example of partial fine-tuning with different learning rates.

18. Transfer Learning Decision Process#

A practical approach is:

Architecture & Data Flow
Do I have a pretrained model?
          |
        Yes
          |
          v
Is source and target data reasonably related?
          |
     +----+----+
     |         |
    Yes        No / weakly related
     |              |
     v              v
Use transfer     Consider whether
learning         transfer is useful
     |
     v
How much target data?
     |
 +---+----------------+
 |                    |
Small                Large
 |                    |
 v                    v
Freeze more        Fine-tune more
layers             layers

The correct strategy depends on:

  • Dataset size
  • Similarity between source and target
  • Amount of domain shift
  • Model size
  • Compute available
  • Risk of overfitting

19. Important Terminology#

Transfer Learning#

Reusing knowledge learned from one task/domain for another.

Feature Extraction#

Using pretrained representations while keeping the backbone frozen.

Pretrained Model#

A model whose parameters were learned previously on another dataset/task.

Freezing#

Preventing selected parameters from being updated.

Unfreezing#

Allowing previously frozen parameters to be updated.

Fine-Tuning#

Continuing training of a pretrained model so its parameters adapt to the target task.

Full Fine-Tuning#

Updating essentially all model parameters on the target task.

Domain Adaptation#

Adapting a model to a target domain whose distribution differs from the source domain.

20. Summary#

ConceptSimple meaning
Transfer LearningReuse learned knowledge for a new task
Feature ExtractionFreeze pretrained features and train a new head
Pretrained ModelModel with previously learned weights
Freezing LayersPrevent selected parameters from updating
Unfreezing LayersAllow selected parameters to update
Fine-TuningAdapt pretrained parameters to a target task
Full Fine-TuningUpdate essentially the whole pretrained model
Domain AdaptationAdapt a model to a different target data distribution

21. Quick Recap#

Architecture & Data Flow
TRANSFER LEARNING

Large Source Dataset
        |
        v
   Pretrained Model
        |
        v
   Target Dataset
        |
        +-----------------------+
        |                       |
        v                       v
Feature Extraction         Fine-Tuning
        |                       |
Freeze backbone            Unfreeze some/all
        |                       |
Train new head             Adapt weights
        |                       |
        +-----------+-----------+
                    |
                    v
              Target Model
Architecture & Data Flow
FEATURE EXTRACTION
 -> Backbone frozen
 -> New head trained

FINE-TUNING
 -> Some pretrained layers updated

FULL FINE-TUNING
 -> Essentially all model parameters updated

DOMAIN ADAPTATION
 -> Adapt model from source distribution
    to a different target distribution

One-Line Mental Model#

Mathematical Formulation
Transfer Learning = Don't learn everything from scratch;
                    reuse a pretrained model and adapt it to the new problem.
Knowledge Checkpoint

24. Transfer Learning Checkpoint

Q1.What is the difference between Feature Extraction and Full Fine-Tuning in Transfer Learning?
AIn Feature Extraction, pretrained backbone weights are frozen (`param.requires_grad = False`) and only the new classification head is trained; in Fine-Tuning, backbone weights are also updated.
BFeature Extraction requires training from random weights from scratch.
CFine-Tuning discards the pretrained backbone completely.
DFeature Extraction only works on text data.
Q2.Why is a lower learning rate (e.g. 10x-100x smaller) recommended for pretrained layers during fine-tuning?
ATo prevent catastrophic forgetting of generic visual or linguistic features learned during expensive large-scale pretraining.
BBecause pretrained weights take up more RAM.
CBecause higher learning rates crash CUDA kernels.
DTo force all gradients to zero.
Q3.When should you prefer Feature Extraction over Fine-Tuning for a new downstream dataset?
AWhen the downstream target dataset is small (e.g. few hundred images) and domain-similar to the pretraining dataset, minimizing overfitting risk.
BWhen you have 10 million labeled downstream images.
CWhen the target domain is completely unrelated to the source domain.
DWhen GPU memory is unlimited.
Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson
23. Diffusion Models & Generative Denoising (DDPM)
Next Lesson
25. Parameter-Efficient Fine-Tuning (PEFT, LoRA & QLoRA)