Advanced

22 min read

#PEFT#LoRA#QLoRA#Fine-Tuning#Low-Rank Adaptation#Quantization#LLMs

25. Parameter-Efficient Fine-Tuning (PEFT, LoRA & QLoRA)

State-of-the-art parameter-efficient adaptation: Low-Rank Adaptation (LoRA) matrix decomposition, rank selection, alpha scaling, 4-bit NormalFloat QLoRA, and Prefix Tuning.

Parameter-Efficient Fine-Tuning: Complete Notes (Beginner to Advanced)

1. Parameter-Efficient Fine-Tuning (PEFT)#

Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques for adapting a large pretrained model to a new task while updating only a small portion of its parameters.

Traditional full fine-tuning looks like:

Architecture & Data Flow
Pretrained Model
      |
      v
Update most/all parameters
      |
      v
Fine-Tuned Model

PEFT instead keeps most pretrained parameters frozen:

Architecture & Data Flow
Pretrained Model
      |
      +--------------------+
      |                    |
Frozen Parameters     Small Trainable
                      Parameters
                           |
                           v
                    Adapted Model

Why PEFT?#

Large models can contain millions or billions of parameters.

Updating all of them can require:

Large GPU memory
Large optimizer states
More storage for each fine-tuned model
More computation
Greater risk of overfitting on small datasets

PEFT reduces the number of trainable parameters while retaining most of the pretrained model.

Basic Idea#

Architecture & Data Flow
Large Pretrained Model
        |
        | freeze most parameters
        v
Small trainable adaptation
        |
        v
Target task

The original model weights remain available, while the learned PEFT parameters contain the task-specific adaptation.

2. LoRA#

LoRA stands for Low-Rank Adaptation.

LoRA adapts a pretrained model by freezing the original weight matrix and learning a low-rank update instead of directly updating the full matrix.

Suppose a pretrained layer contains:

›W

Full fine-tuning changes it to:

›W + ΔW

LoRA represents the update as:

Mathematical Formulation
ΔW = B A

where A and B are much smaller matrices.

Therefore:

Mathematical Formulation
W' = W + BA

The original W remains frozen.

Conceptual Architecture#

Architecture & Data Flow
                 Input x
                    |
          +---------+---------+
          |                   |
          v                   v
     Frozen W x          LoRA branch
          |                   |
          |                 A x
          |                   |
          |                 B(Ax)
          |                   |
          +---------+---------+
                    |
                    v
              Output

The LoRA branch learns the task-specific update.

Why Low Rank?#

If:

›W ∈ R^(d_out × d_in)

then full fine-tuning requires:

›d_out × d_in

trainable parameters.

LoRA uses:

›A ∈ R^(r × d_in)
B ∈ R^(d_out × r)

so the number of trainable parameters is:

Mathematical Formulation
r × d_in + d_out × r
=
r(d_in + d_out)

where:

›r << d_in, d_out

This can dramatically reduce the number of trainable parameters.

3. LoRA Scaling#

LoRA commonly uses a scaling factor:

Mathematical Formulation
W' = W + (α/r) BA

where:

W = frozen pretrained weight
A, B = trainable low-rank matrices
r = LoRA rank
α = scaling hyperparameter

The exact implementation can include additional conventions, but the central idea is that the learned update is low-rank.

LoRA Rank#

The rank r controls the size and expressive capacity of the LoRA update.

text
Small r
   |
Fewer parameters
   |
Lower adaptation capacity

Large r
   |
More parameters
   |
Higher adaptation capacity

The best value depends on the model and task.

4. LoRA Initialization#

A common LoRA initialization strategy makes the initial LoRA update effectively zero.

For example:

Mathematical Formulation
A = random initialization
B = zeros

Then initially:

Mathematical Formulation
BA = 0

so:

Mathematical Formulation
W' ≈ W

This allows training to begin from behavior close to the original pretrained model.

5. Where LoRA Is Applied#

LoRA is commonly applied to selected linear transformations inside Transformer models.

For example:

Architecture & Data Flow
Transformer Block
      |
      +-- Query projection
      +-- Key projection
      +-- Value projection
      +-- Output projection
      +-- Feed-forward projections

A common configuration may target:

›Q and V projections

but LoRA can also be applied to other linear layers.

Important Point#

LoRA does not require modifying the conceptual Transformer architecture.

Instead, it adds trainable low-rank adaptation branches to selected existing layers.

6. QLoRA#

QLoRA combines quantization with LoRA to make fine-tuning large language models more memory-efficient.

The central idea is:

Architecture & Data Flow
Quantized pretrained model
          +
       LoRA adapters
          |
          v
Parameter-efficient fine-tuning

A commonly associated QLoRA setup uses:

text
4-bit quantized base model
+
LoRA adapters

The pretrained base weights are stored in a low-bit representation, while the LoRA parameters remain trainable.

Basic Flow#

Architecture & Data Flow
Pretrained LLM
      |
      v
Quantize base weights
      |
      v
4-bit model weights
      |
      +----------------+
                       |
                    LoRA adapters
                       |
                       v
                Train adapters

This significantly reduces memory required to hold the base model during fine-tuning.

7. Why QLoRA Saves Memory#

Consider full fine-tuning of a large model.

You may need memory for:

text
Model weights
+
Gradients
+
Optimizer states
+
Activations

QLoRA reduces the memory required for the frozen base model by storing it in a low-bit format and trains only a small number of additional parameters.

Conceptually:

Architecture & Data Flow
FULL FINE-TUNING

Large weights
+ gradients
+ optimizer states
+ activations
        |
        v
High memory requirement


QLoRA

Quantized frozen weights
+ small LoRA parameters
+ optimizer states for adapters
+ activations
        |
        v
Much lower memory requirement

QLoRA uses additional techniques to make low-bit training practical, including NF4 quantization, double quantization, and paged optimizers in the original approach.

8. NF4 Quantization in QLoRA#

QLoRA introduced NormalFloat4 (NF4), a 4-bit data type designed for quantizing normally distributed pretrained weights.

The main idea is to represent weights using a small number of quantized values while preserving useful information.

Conceptually:

Architecture & Data Flow
FP16 / BF16 weights
        |
        v
     NF4
        |
        v
4-bit representation

This reduces storage and memory requirements for the frozen base model.

Important Point#

Quantization does not mean the model becomes "4-bit in every operation."

The exact storage, computation, dequantization, and hardware behavior depend on the implementation.

10. Adapter Layer Mathematics#

Let the input to an adapter be:

›h

A simple bottleneck adapter can be represented as:

Mathematical Formulation
Adapter(h)
=
W_up σ(W_down h + b_down) + b_up

The adapter output is then combined with the original representation, commonly through a residual connection:

Mathematical Formulation
h' = h + Adapter(h)

where:

›W_down

reduces the dimensionality and:

›W_up

projects it back.

Why This Is Efficient#

If:

Mathematical Formulation
hidden dimension = d
bottleneck dimension = m

and:

›m << d

then the adapter contains far fewer parameters than a full transformation from d to d.

11. Adapter Layers vs LoRA#

Both are PEFT techniques, but they modify the model differently.

Adapter#

Adds a small neural network module:

Architecture & Data Flow
Original representation
        |
        +----> Adapter
        |
        v
      Output

LoRA#

Adds a low-rank update to an existing weight transformation:

Architecture & Data Flow
Original W
   +
Low-rank BA
   |
   v
Adapted W

Comparison#

Feature	Adapter	LoRA
Main idea	Add small trainable modules	Learn low-rank weight updates
Base weights	Frozen	Frozen
Trainable parameters	Small	Small
Adds new layers/modules?	Yes	Adds low-rank branches to selected layers
Common use	Transformer adaptation	LLM fine-tuning

12. Prompt Tuning#

Prompt tuning is a PEFT method where the model's pretrained parameters remain frozen and a small set of learnable prompt embeddings is optimized.

Instead of manually writing:

›"Classify this sentence:"

the model receives learned continuous vectors.

Conceptually:

Architecture & Data Flow
Input tokens
    +
Learnable prompt vectors
    |
    v
Frozen Language Model
    |
    v
Output

Example#

Instead of:

›[Text]

the model may receive:

›[P1] [P2] [P3] [P4] [Text]

where:

›P1, P2, P3, P4

are trainable embedding vectors.

The pretrained model itself remains frozen.

13. Soft Prompts#

Prompt tuning uses soft prompts, which are continuous vectors rather than ordinary human-readable words.

text
Hard prompt:
"Classify the sentiment:"

Soft prompt:
[p1, p2, p3, p4, ...]

The vectors are optimized through gradient descent.

Training Flow#

Architecture & Data Flow
Initialize soft prompt
        |
        v
Frozen language model
        |
        v
Task output
        |
        v
Calculate loss
        |
        v
Update only prompt embeddings
        |
        v
Repeat

This can result in a very small number of trainable parameters compared with updating the model itself.

14. Prefix Tuning#

Prefix tuning is another PEFT method that learns a small set of continuous vectors called a prefix.

These learned vectors are injected into the Transformer as additional conditioning information, commonly through the attention mechanism.

Conceptually:

Architecture & Data Flow
Learned Prefix
      |
      v
Attention layers
      ^
      |
Input tokens

Unlike prompt tuning, which is often described as adding learnable embeddings to the input sequence, prefix tuning typically introduces learned prefix representations into the attention computation across Transformer layers.

Basic Idea#

Architecture & Data Flow
Input tokens
     |
     v
Transformer
     ^
     |
Learned prefix representations

The pretrained model parameters remain frozen.

15. Prompt Tuning vs Prefix Tuning#

These techniques are closely related but operate at different points.

Prompt Tuning#

Learned vectors are attached to the input representation.

Architecture & Data Flow
[Soft Prompt] + [Input Tokens]
              |
              v
       Frozen Transformer

Prefix Tuning#

Learned prefix representations condition the Transformer attention layers.

Architecture & Data Flow
Learned Prefix
      |
      v
Attention computation
      ^
      |
Input representations

Comparison#

Feature	Prompt Tuning	Prefix Tuning
Base model	Frozen	Frozen
Trainable object	Soft prompt embeddings	Learned prefix representations
Main location	Input embedding sequence	Transformer attention layers
Number of trainable parameters	Very small	Very small
Main idea	Learn what prompt vectors should be	Learn attention-level conditioning

The exact implementation details vary across architectures and libraries.

16. PEFT Methods Compared#

Architecture & Data Flow
                    PEFT
                     |
       +-------------+-------------+
       |             |             |
       v             v             v
     LoRA         Adapters      Prompt Methods
       |             |             |
       |             |       +-----+------+
       |             |       |            |
       |             |       v            v
       |             |    Prompt       Prefix
       |             |    Tuning       Tuning
       |
      QLoRA

LoRA#

text
Freeze W
+
Train low-rank update BA

QLoRA#

text
Quantize base W
+
Train LoRA

Adapters#

text
Freeze model
+
Train small bottleneck modules

Prompt Tuning#

text
Freeze model
+
Train soft prompt embeddings

Prefix Tuning#

text
Freeze model
+
Train attention-level prefix representations

17. PEFT vs Full Fine-Tuning#

Full Fine-Tuning#

Architecture & Data Flow
Pretrained Model
      |
      v
Update essentially all parameters
      |
      v
Task-Specific Model

PEFT#

Architecture & Data Flow
Pretrained Model
      |
      +------------------+
      |                  |
  Frozen base       Small trainable
                    PEFT parameters
                         |
                         v
                  Task-specific behavior

Comparison#

Feature	Full Fine-Tuning	PEFT
Base model	Trainable	Mostly frozen
Trainable parameters	Most/all	Small fraction
GPU memory	High	Lower
Per-task storage	Large	Small
Training cost	High	Lower
Adaptation capacity	Very high	Depends on PEFT method/configuration

18. Why PEFT Is Important for Large Language Models#

Suppose a model has:

›70 billion parameters

Full fine-tuning requires updating a huge number of parameters.

With PEFT:

text
70B base parameters
+
small trainable adapter

The base model can be reused across many tasks.

Mathematical Formulation
                 Base Model
                /    |    \
               /     |     \
          Task A   Task B   Task C
            |        |        |
         Adapter   Adapter  Adapter

This allows different task-specific adaptations without storing a complete copy of the entire model for every task.

19. PEFT Storage Concept#

Suppose:

Mathematical Formulation
Base model = very large
Adapter = very small

Instead of storing:

Mathematical Formulation
Model A = full model
Model B = full model
Model C = full model

we can store:

text
Base model
   +
Adapter A
Adapter B
Adapter C

At inference time, the appropriate adapter can be loaded with the shared base model.

This is especially useful when supporting many task-specific or domain-specific versions of a large model.

20. Choosing a PEFT Method#

A simplified decision process is:

Architecture & Data Flow
Need to adapt a large Transformer?
          |
          v
Keep base model mostly frozen
          |
          v
Choose adaptation method
          |
    +-----+------+-------+---------+
    |            |       |         |
   LoRA       Adapter   Prompt    Prefix
    |
  QLoRA

LoRA#

A strong general-purpose choice when you want to modify internal model transformations with relatively few trainable parameters.

QLoRA#

Useful when GPU memory is constrained and the base model is large.

Adapter Layers#

Useful when you want explicit trainable modules inserted into the architecture.

Prompt Tuning#

Useful when extremely few trainable parameters are desired and the model supports effective prompt-based adaptation.

Prefix Tuning#

Useful when conditioning attention layers through learned prefix representations is appropriate.

There is no universally best PEFT method. Performance depends on the base model, task, dataset, target modules, rank/prompt size, and training setup.

21. Simple LoRA Concept in PyTorch#

A simplified LoRA linear layer can be represented as:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()

        self.weight = nn.Parameter(
            torch.randn(out_features, in_features),
            requires_grad=False
        )

        self.A = nn.Parameter(
            torch.randn(rank, in_features) * 0.01
        )

        self.B = nn.Parameter(
            torch.zeros(out_features, rank)
        )

        self.scale = alpha / rank

    def forward(self, x):
        base = x @ self.weight.T
        update = (x @ self.A.T) @ self.B.T

        return base + self.scale * update

This demonstrates the central LoRA idea:

text
Frozen base transformation
+
Trainable low-rank update

A production implementation would normally use pretrained model weights and established PEFT libraries rather than defining the complete mechanism manually.

22. Mental Model of PEFT#

Think of a large pretrained model as a large machine.

Full Fine-Tuning#

›Change the entire machine

LoRA#

text
Keep the machine unchanged
+
attach small learned adjustment mechanisms

QLoRA#

text
Keep the machine in a compressed/quantized form
+
attach LoRA adjustments

Adapters#

›Insert small trainable modules into the machine

Prompt Tuning#

›Learn a small set of input vectors
that steer the frozen model

Prefix Tuning#

›Learn internal conditioning vectors
that influence attention

23. Summary#

Concept	Simple meaning
PEFT	Fine-tune a pretrained model by training only a small subset of parameters
LoRA	Learns low-rank updates while freezing original weights
QLoRA	Combines quantized base weights with LoRA adaptation
Adapter Layers	Small trainable bottleneck modules inserted into a frozen model
Prompt Tuning	Learns soft prompt embeddings while keeping the model frozen
Prefix Tuning	Learns attention-level prefix representations while keeping the model frozen

24. Quick Recap#

Architecture & Data Flow
PEFT
 |
 +-- LoRA
 |     -> Low-rank weight updates
 |
 +-- QLoRA
 |     -> Quantized base model + LoRA
 |
 +-- Adapter Layers
 |     -> Small trainable bottleneck modules
 |
 +-- Prompt Tuning
 |     -> Learnable soft prompt embeddings
 |
 +-- Prefix Tuning
       -> Learnable attention-level prefixes

Architecture & Data Flow
FULL FINE-TUNING
    |
    v
Update most/all model parameters

PEFT
    |
    v
Freeze most model parameters
    |
    v
Train small adaptation parameters

One-Line Mental Model#

Mathematical Formulation
PEFT = Keep the expensive pretrained model mostly frozen
       and learn a small set of parameters that adapts it to the new task.

Knowledge Checkpoint

25. PEFT & LoRA Checkpoint

Q1.How does Low-Rank Adaptation (LoRA) parameterize weight updates during fine-tuning?

AIt freezes the base weight matrix W_0 (d x k) and adds a low-rank decomposition delta_W = B · A, where B (d x r) and A (r x k) with rank r << min(d, k).

BIt deletes 90% of base weights randomly.

CIt quantizes weights to 1-bit integers.

DIt adds new transformer layers on top of the model.

Q2.Why is matrix A initialized with Gaussian noise while matrix B is initialized to exact zeros in LoRA?

ASo that delta_W = B · A equals exactly 0 at step 0, ensuring model behavior at the start of fine-tuning is identical to the original pretrained model.

BTo prevent CUDA out-of-memory errors.

CBecause zeros are not allowed in matrix A.

DTo disable gradient backpropagation.

Q3.What three major innovations make QLoRA capable of fine-tuning 65B LLMs on a single 48GB GPU?

A4-bit NormalFloat (NF4) data type, Double Quantization of quantization constants, and Paged Optimizers for GPU memory spike handling.

BDiscarding attention heads, removing MLP layers, and reducing context length to 10 tokens.

CConverting PyTorch code to JavaScript.

DUsing CPU disk swap exclusively.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

24. Transfer Learning & Domain Adaptation

Next Lesson

26. Deep Learning Frameworks (PyTorch Core Architecture)