Advanced

180–240 min read

#Fine-Tuning#Supervised Fine-Tuning#Instruction Tuning#Hugging Face#LoRA#QLoRA#PEFT#Adapters#Quantization#Datasets#Model Customization#Catastrophic Forgetting

Fine-Tuning, LoRA, QLoRA & PEFT: Customizing Generative AI Models

A practical beginner-to-advanced guide to adapting foundation models with supervised fine-tuning and parameter-efficient methods such as LoRA and QLoRA, including dataset preparation, training, evaluation, failure modes, model selection, deployment, and the tradeoffs between prompting, RAG, and fine-tuning.

Fine-Tuning, LoRA, QLoRA & PEFT: Customizing Generative AI Models

1. Introduction#

Foundation models are trained on broad datasets and can perform many general-purpose tasks.

But enterprise applications often require specialized behavior.

Examples:

text
Company-specific writing style
Domain-specific terminology
Structured response formats
Specialized classification
Instruction following
Task-specific reasoning patterns

A common progression is:

Architecture & Data Flow
Prompting
 |
 v
Few-shot prompting
 |
 v
RAG
 |
 v
Fine-tuning

These techniques solve different problems.

The central question is not:

›"Can I fine-tune this model?"

It is:

›"Is fine-tuning the right way to improve this application?"

2. Learning Objectives

By the end of this notebook, you should understand:

Why fine-tuning is needed
Pre-training vs fine-tuning
Instruction tuning
Supervised fine-tuning
Dataset preparation
Data quality
Training formats
Chat templates
Tokenization
Training and validation splits
Full fine-tuning
Parameter-efficient fine-tuning
LoRA
QLoRA
PEFT
Adapters
Quantization
Fine-tuning hyperparameters
Learning rate
Batch size
Epochs
Gradient accumulation
Checkpointing
Evaluation
Catastrophic forgetting
Overfitting
Fine-tuning vs RAG
Fine-tuning vs prompting
Fine-tuning multimodal models
Hugging Face training workflow
Model evaluation
Deployment
Fine-tuning projects

3. What Is Fine-Tuning?

Fine-tuning continues training a pre-trained model on a smaller, task-specific dataset.

Conceptually:

Architecture & Data Flow
Pre-trained model
 |
 v
Domain / task dataset
 |
 v
Fine-tuning
 |
 v
Customized model

The original model already contains broad capabilities.

Fine-tuning adjusts model parameters so the model behaves better for a particular distribution of tasks.

4. Pre-Training vs Fine-Tuning

Pre-training#

The model learns broad patterns from a very large dataset.

Architecture & Data Flow
Huge dataset
 |
 v
Training
 |
 v
Foundation model

Fine-tuning#

The model is adapted using a smaller specialized dataset.

Architecture & Data Flow
Foundation model
 |
 v
Specialized dataset
 |
 v
Fine-tuned model

Pre-training creates broad capabilities.

Fine-tuning specializes behavior.

5. Instruction Tuning

Instruction tuning trains a model on examples of:

text
Instruction
+
Input
+
Desired response

Example:

text
Instruction:
Classify the sentiment.

Input:
"The product is excellent."

Response:
positive

Repeated examples teach the model how to follow a task-oriented instruction format.

6. Supervised Fine-Tuning

Supervised Fine-Tuning (SFT) uses examples where the desired output is known.

Conceptually:

Architecture & Data Flow
Input X
 |
 v
Model
 |
 v
Prediction Y'
 |
 v
Compare with Y
 |
 v
Loss
 |
 v
Update parameters

The goal is to reduce the difference between:

›model output

and:

›target output

7. Fine-Tuning Does Not Start From Zero

A common misconception is:

Mathematical Formulation
Fine-tuning = training a model from scratch

Usually:

Mathematical Formulation
Fine-tuning
=
starting from an existing pre-trained model
+
continuing training on specialized data

This is much less computationally expensive than pre-training a foundation model.

8. When Fine-Tuning Makes Sense

Fine-tuning can be useful when you want to change:

text
Behavior
Style
Task performance
Instruction following
Output patterns
Domain-specific task behavior

Examples:

text
Medical report formatting
Legal document classification
Customer-support style
Code transformation
Specialized extraction

For changing factual knowledge that changes frequently, RAG may be more appropriate.

9. Fine-Tuning vs Prompting

Prompting:

text
Model
+
Instructions

No model weights are changed.

Fine-tuning:

Architecture & Data Flow
Model
+
Training examples
 |
 v
Updated weights

Use prompting when the task can be solved through instructions.

Use fine-tuning when repeated examples show that the model needs persistent behavioral adaptation.

10. Fine-Tuning vs RAG

RAG provides external information at inference time.

Architecture & Data Flow
Question
 |
 v
Retriever
 |
 v
Documents
 |
 v
LLM

Fine-tuning changes model behavior:

Architecture & Data Flow
Training data
 |
 v
Fine-tuning
 |
 v
Updated model

A useful rule:

Architecture & Data Flow
Need new knowledge?
 -> RAG

Need new behavior?
 -> Fine-tuning

This is a simplification, but it is a useful starting point.

11. Fine-Tuning and RAG Together

They can be combined.

Architecture & Data Flow
Fine-tuned model
 +
RAG
 |
 v
Specialized application

For example:

text
Fine-tuning:
Teach customer-support response style.

RAG:
Provide current company policies.

This separates:

›Behavior

from:

›External knowledge

12. Dataset Quality Is Critical

A small amount of excellent data can be more valuable than a large amount of noisy data.

Bad dataset:

text
Incorrect answers
Inconsistent formatting
Duplicate examples
Conflicting instructions
Low-quality labels

Good dataset:

text
Accurate
Consistent
Representative
Diverse
Well-labeled
Relevant

Fine-tuning amplifies patterns present in the training data.

13. Garbage In, Garbage Out

If training examples contain:

›Wrong classifications

the model can learn those wrong classifications.

If examples contain:

›Inconsistent output formats

the model may produce inconsistent outputs.

Fine-tuning is not a substitute for data quality.

14. Dataset Composition

A useful dataset can include:

text
Common examples
+
Difficult examples
+
Edge cases
+
Negative examples
+
Boundary cases

Avoid building a dataset entirely from easy examples.

15. Training, Validation and Test Sets

Split data chronologically or randomly depending on the problem, while preventing leakage.

Typical structure:

Architecture & Data Flow
Dataset
 |
 +--> Training
 |
 +--> Validation
 |
 +--> Test

The training set updates model parameters.

The validation set helps select training configurations.

The test set estimates final performance.

16. Data Leakage

Leakage occurs when information from evaluation data influences training.

Example:

text
Training:
Customer A's exact test answer

Test:
Customer A's same question

This can make performance appear artificially high.

Keep evaluation data isolated.

17. Deduplication

Duplicate or near-duplicate examples can distort evaluation.

Example:

text
Train:
"What is RAG?"

Test:
"What is RAG?"

or:

text
Train:
"Explain retrieval augmented generation."

Test:
"Explain retrieval-augmented generation."

Use deduplication and similarity checks where appropriate.

18. Data Diversity

Suppose you are fine-tuning a support model.

Include:

text
Simple questions
Complex questions
Angry users
Confused users
Technical issues
Billing issues
Account issues
Ambiguous requests

The model should learn the target task distribution rather than memorize one narrow pattern.

19. Chat Dataset Format

A common conceptual format is:

json
{
 "messages": [
 {
 "role": "system",
 "content": "You are a support assistant."
 },
 {
 "role": "user",
 "content": "I cannot reset my password."
 },
 {
 "role": "assistant",
 "content": "Let's help you reset your password."
 }
 ]
}

The exact schema depends on the model and training framework.

20. Chat Templates

Different models may expect different formatting.

Conceptually:

text
System
User
Assistant

may be converted into a model-specific template:

text
<special tokens>
system content
<special tokens>
user content
<special tokens>
assistant content

Use the tokenizer's supported chat template when available.

Do not invent formatting blindly.

21. Tokenization

Training operates on tokens rather than raw strings.

Architecture & Data Flow
Text
 |
 v
Tokenizer
 |
 v
Token IDs
 |
 v
Model

Example:

›"Hello world"

may become:

›[15496, 995]

The exact IDs depend on the tokenizer.

22. Token Budget

Training cost depends heavily on token count.

Approximate dataset size:

text
Number of examples
×
Average tokens per example

For example:

Mathematical Formulation
100,000 examples
×
500 tokens
=
50,000,000 tokens

Long examples increase training cost.

23. Sequence Length

The model processes sequences up to a supported context length.

Example:

Mathematical Formulation
Sequence length = 2,048 tokens

Longer training examples may need:

text
Truncation
Packing
Chunking

Avoid silently truncating important information.

24. Label Masking

For conversational SFT, you may want the loss to focus on assistant responses.

Conceptually:

Architecture & Data Flow
System tokens -> ignore
User tokens -> ignore
Assistant tokens -> calculate loss

This is commonly called response-only or completion-only loss.

The exact implementation depends on the training framework.

25. Full Fine-Tuning

Full fine-tuning updates most or all model parameters.

Architecture & Data Flow
Model
 |
 +--> Parameter 1
 +--> Parameter 2
 +--> Parameter 3
 +--> ...
 |
 v
Update many parameters

Advantages:

Maximum adaptation capacity
Straightforward conceptual model

Disadvantages:

High GPU memory requirements
Higher compute cost
Larger checkpoints
Greater risk of catastrophic forgetting

26. Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning (PEFT) updates only a small subset or additional set of parameters.

Architecture & Data Flow
Base model
 |
 +--> Frozen parameters
 |
 +--> Trainable parameters

This reduces:

text
Memory
Compute
Storage

LoRA is one of the most popular PEFT methods.

27. LoRA

LoRA stands for Low-Rank Adaptation.

Instead of directly updating a large weight matrix:

›W

LoRA learns a low-rank update:

Mathematical Formulation
W' = W + ΔW

where:

Mathematical Formulation
ΔW = B A

with smaller matrices:

›A
B

The base weight matrix can remain frozen.

28. LoRA Intuition

Suppose:

Mathematical Formulation
W = 4096 × 4096

Updating every parameter is expensive.

LoRA can represent the update using:

Mathematical Formulation
A = 16 × 4096
B = 4096 × 16

The number of trainable parameters becomes much smaller.

The exact rank is a hyperparameter.

29. LoRA Equation

The original layer:

Mathematical Formulation
y = Wx

becomes:

Mathematical Formulation
y = Wx + BAx

where:

Mathematical Formulation
W = frozen
A = trainable
B = trainable

This lets the model learn a task-specific update without modifying the original matrix directly.

30. LoRA Rank

The rank is often written as:

›r

Small rank:

Mathematical Formulation
r = 4
r = 8
r = 16

Larger rank:

Mathematical Formulation
r = 32
r = 64

Increasing rank increases adaptation capacity and trainable parameters.

Higher is not automatically better.

31. LoRA Alpha

LoRA commonly includes a scaling factor.

Conceptually:

Mathematical Formulation
output =
W x
+
(alpha / r) B A x

The exact implementation may include additional conventions.

The important idea is that LoRA updates are scaled relative to the base layer.

32. LoRA Dropout

LoRA implementations may support dropout on the adaptation path.

Purpose:

›Regularization

It can help reduce overfitting in some settings.

33. Which Layers Should LoRA Modify?

Common target modules include attention projections such as:

text
q_proj
k_proj
v_proj
o_proj

Some configurations also target feed-forward projections.

The appropriate target modules depend on:

text
Model architecture
Task
Memory budget
Training framework

34. QLoRA

QLoRA combines:

text
Quantized base model
+
LoRA adapters

Conceptually:

Architecture & Data Flow
Base model
 |
 v
4-bit quantization
 |
 v
Frozen quantized model
 +
Trainable LoRA adapters

This can significantly reduce memory requirements during fine-tuning.

35. Why QLoRA Is Useful

Large models may not fit comfortably into GPU memory when fully loaded at higher precision.

Quantization reduces the memory footprint.

Then LoRA keeps trainable parameters small.

Architecture & Data Flow
Quantization -> reduce base model memory

LoRA -> reduce trainable parameters

Together:

›QLoRA

can make adaptation more accessible.

36. Quantization

Quantization represents numerical values with lower precision.

Conceptually:

Architecture & Data Flow
FP32
 |
 v
FP16 / BF16
 |
 v
INT8
 |
 v
INT4

Lower precision can reduce:

text
Memory
Bandwidth
Sometimes inference cost

But it may introduce accuracy or compatibility tradeoffs.

37. Quantization Is Not Only for Fine-Tuning

Quantization can be used for:

text
Inference
+
Fine-tuning workflows

For example:

›Quantized model

may be useful for deployment.

QLoRA specifically combines quantized base weights with trainable adapters.

38. Adapters

Adapters add small trainable modules to a frozen model.

Conceptually:

Architecture & Data Flow
Base model
 |
 +--> Frozen layers
 |
 +--> Adapter
 |
 v
Output

Different tasks can use different adapters.

Example:

Architecture & Data Flow
Base model
 |
 +--> Finance adapter
 +--> Support adapter
 +--> Legal adapter

This can be more storage-efficient than keeping a separate full model for every task.

39. PEFT Mental Model

Think:

Architecture & Data Flow
Large base model
 |
 | frozen
 v
Small trainable component
 |
 v
Task-specific behavior

Methods include:

text
LoRA
Adapters
Other parameter-efficient techniques

40. Fine-Tuning Hyperparameters

Important hyperparameters include:

text
Learning rate
Batch size
Epochs
Gradient accumulation
Sequence length
Warmup
Weight decay
LoRA rank
LoRA alpha
LoRA dropout

These strongly affect results.

41. Learning Rate

Learning rate controls the size of parameter updates.

Too high:

text
Training instability
Catastrophic changes
Poor convergence

Too low:

›Very slow learning
Insufficient adaptation

Fine-tuning often uses a relatively small learning rate compared with training from scratch.

42. Batch Size

Batch size is the number of examples processed before a gradient update.

Larger batch:

›More memory
Potentially more stable gradients

Smaller batch:

›Lower memory
More gradient noise

GPU memory often constrains batch size.

43. Gradient Accumulation

If the GPU can process only:

Mathematical Formulation
Batch size = 2

you can accumulate gradients over multiple steps.

Example:

text
2 examples
+
2 examples
+
2 examples
+
2 examples

before updating parameters.

Approximate effective batch size:

text
micro batch size
×
gradient accumulation steps

44. Epochs

One epoch means one pass through the training dataset.

Too few:

›Underfitting

Too many:

›Overfitting

For fine-tuning, more epochs are not necessarily better.

Monitor validation performance.

45. Warmup

Learning-rate warmup starts training with a smaller learning rate and gradually increases it.

Conceptually:

Architecture & Data Flow
Learning rate
 ^
 | ________
 | /
 | /
 |____/
 +----------------> Steps

Warmup can improve training stability.

46. Weight Decay

Weight decay is a regularization mechanism.

It can discourage overly large parameter changes.

It should be chosen based on the training configuration rather than copied blindly from another model.

47. Gradient Clipping

Gradient clipping limits excessively large gradients.

Conceptually:


🐍 PythonInteractive WebAssembly
clip_grad_norm_(parameters, max_norm)

This can help with training stability.

48. Checkpoints

During training, save checkpoints.

text
Step 100
Step 200
Step 300
Step 400

This allows you to:

Resume training
Compare checkpoints
Recover from failures
Select the best checkpoint

49. Early Stopping

Monitor validation performance.

Example:

Architecture & Data Flow
Epoch 1 -> validation loss 1.8
Epoch 2 -> 1.5
Epoch 3 -> 1.2
Epoch 4 -> 1.3
Epoch 5 -> 1.5

The best checkpoint may be:

›Epoch 3

Continuing training may be overfitting.

50. Overfitting in Fine-Tuning

Fine-tuning can overfit surprisingly quickly on small datasets.

Symptoms:

›Training loss decreases
Validation quality worsens

Possible solutions:

More data
Better data diversity
Lower learning rate
Fewer epochs
Regularization
Smaller LoRA rank
Early stopping

51. Catastrophic Forgetting

Fine-tuning can reduce performance on capabilities that were strong before adaptation.

Example:

Mathematical Formulation
Before:
General reasoning = strong

After specialized fine-tuning:
Specialized task = strong
General reasoning = weaker

This is one reason evaluation should include:

text
Target task tests
+
General capability tests

52. Mitigating Catastrophic Forgetting

Possible approaches:

text
More diverse data
+
Lower learning rate
+
Fewer epochs
+
Mixed training data
+
Parameter-efficient tuning

The best method depends on the task.

53. Fine-Tuning Data Mixing

Suppose your target dataset contains:

›90% specialized examples
10% general examples

You may include carefully selected general examples to preserve broader behavior.

This creates a balance between:

text
Specialization
+
General capability

54. Fine-Tuning Workflow

A practical workflow:

Architecture & Data Flow
Define task
 |
 v
Collect data
 |
 v
Clean data
 |
 v
Format examples
 |
 v
Train/validation/test split
 |
 v
Choose base model
 |
 v
Choose full FT or PEFT
 |
 v
Configure training
 |
 v
Train
 |
 v
Evaluate
 |
 v
Error analysis
 |
 v
Iterate
 |
 v
Deploy

55. Start With a Baseline

Before fine-tuning:

›Evaluate base model

Then compare:

text
Base model
vs
Prompted base model
vs
RAG
vs
Fine-tuned model

Without a baseline, you cannot know whether fine-tuning helped.

56. Evaluation Categories

A good evaluation suite includes:

text
Task performance
General capabilities
Safety
Instruction following
Structured output
Robustness
Latency
Cost

For domain applications, add domain-specific metrics.

57. Classification Fine-Tuning

Fine-tuning can be used for classification.

Example:

text
Input:
"My card was charged twice."

Output:
billing

Dataset:

›text -> label

Evaluate with:

text
Accuracy
Precision
Recall
F1
Confusion matrix

58. Structured Extraction Fine-Tuning

Example:

text
Input:
Invoice text

Output:
{
 "invoice_number": "...",
 "total": 1250,
 "currency": "USD"
}

Evaluate:

text
Field accuracy
Schema validity
Exact field match

59. Style Fine-Tuning

Fine-tuning can teach a consistent style.

Example:

text
Formal customer support
Technical documentation
Brand-specific writing

However, style can often be achieved with prompting.

Fine-tuning is more compelling when the desired behavior must be persistent and consistent across many prompts.

60. Tool-Calling Fine-Tuning

Training examples can teach a model to produce tool calls.

Conceptually:

text
User:
What is 20% of 500?

Assistant:
tool_call(calculator, ...)

The dataset should include:

text
Correct tool
Correct arguments
Correct response after tool result

Evaluate both tool selection and argument correctness.

61. Fine-Tuning for Agents

Agent fine-tuning is more complicated because the desired behavior can involve trajectories.

Example:

Architecture & Data Flow
Question
 |
 v
Search tool
 |
 v
Read result
 |
 v
Calculator
 |
 v
Final answer

Training may require high-quality examples of:

text
Tool selection
Tool arguments
Reasoning/action structure
Final responses

Use careful evaluation because incorrect trajectories can teach unsafe behavior.

62. Fine-Tuning Multimodal Models

Multimodal fine-tuning can adapt:

›Image + text -> response

or:

›Image -> structured extraction

or:

›Audio + text -> response

or:

›Video + text -> response

The same principles apply:

text
High-quality data
+
Correct formatting
+
Evaluation

But the dataset may be much more expensive to create.

63. Multimodal Dataset Example

Conceptually:

json
{
 "messages": [
 {
 "role": "user",
 "content": [
 {
 "type": "image",
 "path": "product.jpg"
 },
 {
 "type": "text",
 "text": "Identify visible damage."
 }
 ]
 },
 {
 "role": "assistant",
 "content": "A crack is visible near the upper-right corner."
 }
 ]
}

The exact schema depends on the model.

64. Hugging Face Ecosystem

A common open-source fine-tuning stack can include:

text
Transformers
Datasets
PEFT
Accelerate
bitsandbytes
TRL

These components solve different problems.

Conceptually:

Architecture & Data Flow
Datasets
 |
 v
Transformers
 |
 +--> PEFT
 |
 +--> Quantization
 |
 v
Trainer / training loop

APIs evolve, so verify compatibility between package versions and the selected model.

65. Loading a Dataset

Conceptually:


🐍 PythonInteractive WebAssembly
from datasets import load_dataset

dataset = load_dataset(
 "json",
 data_files="train.jsonl"
)

Then inspect:


🐍 PythonInteractive WebAssembly
print(dataset)

Always validate the dataset before training.

66. Inspecting Examples


🐍 PythonInteractive WebAssembly
print(dataset["train"][0])

Check:

text
Roles
Content
Missing fields
Unexpected values
Length
Formatting

Never begin a long training job before inspecting samples.

67. Tokenization Inspection

A useful debugging step:


🐍 PythonInteractive WebAssembly
tokens = tokenizer(
 dataset["train"][0]["text"]
)

print(tokens["input_ids"][:20])

Also inspect:

text
Token count
Special tokens
Truncation
Padding

68. Length Distribution

Measure example lengths.

Conceptually:


🐍 PythonInteractive WebAssembly
lengths = [
 len(tokenizer(example["text"])["input_ids"])
 for example in dataset["train"]
]

Then analyze:

text
Minimum
Median
Mean
P95
Maximum

This helps choose:

›max sequence length

69. Training Configuration

A training configuration may specify:

text
output directory
learning rate
batch size
epochs
evaluation strategy
save strategy
logging
gradient accumulation
precision

Example:


🐍 PythonInteractive WebAssembly
training_args = {
 "learning_rate": 2e-5,
 "num_train_epochs": 3,
 "per_device_train_batch_size": 2,
}

The exact training API depends on the framework version.

70. LoRA Configuration

Conceptually:


🐍 PythonInteractive WebAssembly
from peft import LoraConfig

lora_config = LoraConfig(
 r=16,
 lora_alpha=32,
 lora_dropout=0.05,
 target_modules=[
 "q_proj",
 "v_proj"
 ]
)

The correct target modules depend on the architecture.

71. Applying LoRA

Conceptually:


🐍 PythonInteractive WebAssembly
from peft import get_peft_model

model = get_peft_model(
 model,
 lora_config
)

Then inspect trainable parameters.


🐍 PythonInteractive WebAssembly
model.print_trainable_parameters()

You should verify that only the intended parameters are trainable.

72. Quantized Loading

For QLoRA-style workflows, the base model may be loaded using an appropriate quantization configuration.

Conceptually:


🐍 PythonInteractive WebAssembly
quantization_config = ...

Then:


🐍 PythonInteractive WebAssembly
model = load_model(
 quantization_config=quantization_config
)

The exact configuration depends on:

text
Model architecture
Transformers version
Quantization backend
GPU support

73. Training

A conceptual training loop:


🐍 PythonInteractive WebAssembly
for batch in train_loader:

 outputs = model(
 input_ids=batch["input_ids"],
 labels=batch["labels"]
 )

 loss = outputs.loss

 loss.backward()

 optimizer.step()
 optimizer.zero_grad()

Framework trainers automate many of these details.

Understanding the loop remains useful for debugging.

74. Loss

For causal language modeling, the model predicts the next token.

Conceptually:

Architecture & Data Flow
Token 1 -> predict Token 2
Token 2 -> predict Token 3
Token 3 -> predict Token 4

Training minimizes:

›Cross-entropy loss

over the target tokens.

75. Perplexity

Perplexity is related to language-model loss.

Conceptually:

Mathematical Formulation
perplexity = exp(loss)

Lower perplexity can indicate better token-level prediction.

However:

›Lower perplexity

does not automatically mean:

›Better task performance

Always evaluate the actual task.

76. Why Loss Is Not Enough

Two models may have similar training loss but different:

text
Instruction following
Correctness
Safety
Tool use
Structured output

Therefore:

text
Training loss
+
Task evaluation

should be considered together.

77. Error Analysis

After evaluation, inspect failures.

Categorize them:

text
Knowledge failure
Reasoning failure
Formatting failure
Instruction failure
Hallucination
Tool failure
Data ambiguity

Then improve the dataset or training setup accordingly.

78. Data-Centric Iteration

A powerful loop is:

Architecture & Data Flow
Train
 |
 v
Evaluate
 |
 v
Find failures
 |
 v
Add representative examples
 |
 v
Train again

The objective is not to blindly increase dataset size.

It is to improve the quality and coverage of the data.

79. Fine-Tuning Failure Modes

Common problems:

Training instability#

Possible causes:

text
Learning rate too high
Bad data
Numerical issues

Overfitting#

›Training improves
Validation worsens

Underfitting#

›Training and validation both poor

Catastrophic forgetting#

›Specialized task improves
General capability declines

80. Data Imbalance

Suppose:

›90% billing examples
10% technical examples

The model may become biased toward billing behavior.

Measure:

›Per-category performance

not just overall performance.

81. Synthetic Data

Synthetic examples can expand a dataset.

Conceptually:

Architecture & Data Flow
Human examples
 |
 v
LLM generates candidate examples
 |
 v
Filtering / validation
 |
 v
Training dataset

Do not automatically trust synthetic data.

Use:

text
Validation
Deduplication
Human review
Quality filters

82. Distillation

Knowledge distillation can train a smaller model to reproduce useful behavior from a larger model.

Architecture & Data Flow
Large teacher
 |
 v
Generated examples / signals
 |
 v
Smaller student

This can help with:

text
Latency
Cost
Local deployment

Distillation and fine-tuning can also be combined.

83. Model Selection

Choose the base model based on:

text
Task capability
Language support
Context length
Tool calling
Multimodal support
Model size
Hardware requirements
License
Deployment constraints
Quality
Cost

The largest model is not always the best model.

84. Hardware Planning

Training requirements depend on:

text
Model size
Precision
Sequence length
Batch size
Optimizer
Gradient checkpointing
LoRA vs full fine-tuning

A large model with LoRA may fit on hardware where full fine-tuning does not.

85. Gradient Checkpointing

Gradient checkpointing trades:

›More computation

for:

›Less activation memory

This can allow longer sequences or larger models under limited GPU memory.

86. Mixed Precision

Training may use:

›FP16
BF16

to reduce memory and improve throughput on supported hardware.

BF16 can be particularly useful on compatible modern accelerators because of its wider exponent range.

Hardware support matters.

87. Distributed Training

For larger models, training can be distributed across multiple GPUs.

Conceptually:

Architecture & Data Flow
GPU 1
GPU 2
GPU 3
GPU 4
 |
 v
Distributed training

Strategies include:

text
Data parallelism
Tensor parallelism
Pipeline parallelism
Sharding

The appropriate strategy depends on model size and infrastructure.

88. Adapter Deployment

With LoRA, you may store:

text
Base model
+
Small adapter

instead of a complete copy of the model.

This enables:

text
One base model
+
Many adapters

Example:

Architecture & Data Flow
Base model
 |
 +--> Finance adapter
 +--> Support adapter
 +--> Legal adapter

89. Merging LoRA Adapters

Adapters can sometimes be merged into the base model.

Conceptually:

Architecture & Data Flow
Base weights
+
LoRA update
 |
 v
Merged model

Benefits can include simpler deployment.

Tradeoffs include:

text
Loss of adapter flexibility
Larger artifact
Potential compatibility considerations

90. Serving Fine-Tuned Models

A production architecture may look like:

Architecture & Data Flow
Client
 |
 v
API
 |
 v
Inference server
 |
 v
Base model + adapter
 |
 v
Response

For higher-scale systems, use an inference engine designed for efficient LLM serving.

91. Fine-Tuning vs Separate Models

Suppose you need:

text
Finance assistant
Support assistant
Legal assistant

Options include:

Separate full models#

text
Model A
Model B
Model C

Shared base + adapters#

Architecture & Data Flow
Base
 |
 +--> Finance adapter
 +--> Support adapter
 +--> Legal adapter

The adapter approach can greatly reduce storage and deployment duplication.

92. Fine-Tuning Governance

Enterprise fine-tuning should track:

text
Dataset version
Model version
Training configuration
Code version
Adapter version
Evaluation results
Approval status
Deployment version

This is essential for reproducibility.

93. Security of Fine-Tuning Data

Training data can contain sensitive information.

Before training:

Architecture & Data Flow
Detect sensitive data
 |
 v
Redact / approve
 |
 v
Training dataset

Consider:

PII
Confidential documents
Secrets
Customer data
Proprietary source code

Do not assume training data is safe simply because it is internal.

94. Memorization Risk

Fine-tuning can increase the chance that sensitive examples influence model behavior.

Reduce unnecessary exposure through:

text
Data minimization
Deduplication
Redaction
Careful dataset construction
Evaluation for memorization

Sensitive data should be handled according to organizational policy.

95. Fine-Tuning Decision Tree

Start with:

›What problem are we solving?

If:

›Current information

consider:

›RAG

If:

›Prompt instructions are sufficient

use:

›Prompting

If:

›Persistent task behavior needs improvement

consider:

›Fine-tuning

If:

›Full fine-tuning is too expensive

consider:

›PEFT / LoRA

If:

›GPU memory is constrained

consider:

›QLoRA

96. Fine-Tuning Experiment Matrix

Run controlled experiments.

Example:

text
 Base Prompt RAG LoRA QLoRA
Task accuracy 82 88 93 95 94
Safety 98 98 98 97 97
Latency ...
Cost ...

The best solution is the one that meets the application's requirements.

97. Experiment Tracking

Track every experiment:

text
Experiment ID
Base model
Dataset version
Prompt
LoRA rank
Learning rate
Epochs
Batch size
Hardware
Validation score
Test score
Cost
Notes

This prevents repeated experiments and makes decisions auditable.

98. Practical Project 1: Sentiment Classifier

Build a small SFT dataset:

›Input -> sentiment

Compare:

text
Prompting
vs
Fine-tuning

Evaluate:

text
Accuracy
Precision
Recall
F1

99. Practical Project 2: Support Intent Model

Create categories:

text
billing
account
technical
shipping
other

Fine-tune a model to classify requests.

Add:

text
Ambiguous examples
Edge cases
Out-of-domain examples

Evaluate per category.

100. Practical Project 3: Structured Invoice Extraction

Create examples:

Architecture & Data Flow
Invoice text/image
 |
 v
Structured JSON

Fine-tune a suitable model.

Measure:

text
Field accuracy
Schema validity
Missing-field rate

Compare against prompting.

101. Practical Project 4: LoRA Domain Adapter

Choose a domain such as:

text
Technical support
Finance
Legal-style document processing

Create a curated dataset.

Train:

›Base model + LoRA

Compare:

text
Base model
Prompted model
LoRA model

Measure quality, memory usage, latency, and cost.

102. Practical Project 5: QLoRA Experiment

Choose a model that can be reasonably adapted on your available hardware.

Compare:

text
LoRA
vs
QLoRA

Track:

text
GPU memory
Training time
Validation quality
Final quality
Checkpoint size

Document the tradeoffs.

103. Advanced Exercise: Catastrophic Forgetting

Evaluate the base model on:

text
General benchmark set
+
Target task set

Fine-tune.

Evaluate again.

Compare:

›General performance
Target performance

Determine whether specialization caused unacceptable regression.

104. Advanced Exercise: Data Quality

Create two datasets:

text
Dataset A:
Large but noisy

Dataset B:
Smaller but carefully curated

Fine-tune separate adapters.

Compare results.

This demonstrates:

›Data quality can matter more than raw dataset size.

105. Advanced Exercise: Rank Selection

Train LoRA adapters with:

Mathematical Formulation
r = 4
r = 8
r = 16
r = 32

Compare:

text
Quality
Trainable parameters
Training time
Memory

Determine whether increasing rank produces meaningful improvement.

106. Advanced Exercise: RAG vs Fine-Tuning

Create a domain task requiring:

text
Stable response behavior
+
Changing external knowledge

Compare:

text
Prompt only
Fine-tuning only
RAG only
Fine-tuning + RAG

Document which component solves which problem.

107. Advanced Exercise: Adapter Routing

Build:

Architecture & Data Flow
Base model
 |
 +--> Finance adapter
 +--> Support adapter
 +--> Technical adapter

Create a router that selects the adapter based on task type.

Evaluate:

text
Routing accuracy
Task performance
Latency

108. Common Mistakes

Mistake 1: Fine-tuning before building a baseline#

You may solve the wrong problem.

Mistake 2: Using low-quality data#

The model learns the wrong patterns.

Mistake 3: Too many epochs#

This can cause overfitting.

Mistake 4: Treating training loss as final quality#

Task-level evaluation matters.

Mistake 5: Ignoring general capability regression#

Fine-tuning can cause forgetting.

Mistake 6: Fine-tuning frequently changing knowledge#

RAG may be a better solution.

Mistake 7: Using full fine-tuning when PEFT is sufficient#

This can waste substantial compute and memory.

109. Complete Fine-Tuning Architecture

Architecture & Data Flow
 TRAINING DATA
 |
 v
 Data Validation
 |
 v
 Dataset Formatting
 |
 v
 Train / Val / Test
 |
 v
 Base Foundation Model
 |
 +-------------+-------------+
 | |
 v v
 Full Fine-Tuning PEFT
 | |
 | +------+------+
 | | |
 | LoRA QLoRA
 | | |
 +--------------------+-------------+
 |
 v
 Evaluation
 |
 +-------------+-------------+
 | |
 v v
 Target Task Tests General Capability
 | |
 +-------------+-------------+
 |
 v
 Validation
 |
 v
 Deployment

110. Production Fine-Tuning Lifecycle

Architecture & Data Flow
Problem Definition
 |
 v
Baseline
 |
 v
Data Collection
 |
 v
Data Quality
 |
 v
Experiment
 |
 v
Evaluation
 |
 v
Security Review
 |
 v
Model Approval
 |
 v
Canary Deployment
 |
 v
Production Monitoring
 |
 v
Continuous Evaluation

Fine-tuning should be treated as an engineering lifecycle, not a one-time training command.

111. Final Mental Model

Think of model customization as a hierarchy:

Architecture & Data Flow
Prompting
 |
 v
Few-shot examples
 |
 v
RAG
 |
 v
Fine-tuning
 |
 +--> Full fine-tuning
 |
 +--> PEFT
 |
 +--> LoRA
 |
 +--> QLoRA

Use the least expensive technique that reliably solves the problem.

The core distinction is:

Architecture & Data Flow
Prompting -> temporary instructions

RAG -> external knowledge at inference time

Fine-tuning -> learned behavioral adaptation

LoRA / PEFT -> efficient behavioral adaptation

QLoRA -> memory-efficient LoRA-style adaptation

112. Key Takeaways

Fine-tuning adapts an existing foundation model.
Instruction tuning teaches models to follow task-oriented instructions.
Supervised fine-tuning requires high-quality examples.
Dataset quality strongly affects results.
Training, validation, and test sets must be separated carefully.
Chat templates and tokenization must match the model.
Full fine-tuning updates many model parameters.
PEFT reduces the number of trainable parameters.
LoRA learns low-rank updates while keeping the base model largely frozen.
QLoRA combines quantized base weights with LoRA adapters.
Quantization reduces memory requirements but introduces tradeoffs.
Learning rate, batch size, epochs, and sequence length strongly affect training.
Overfitting is a major risk with small fine-tuning datasets.
Catastrophic forgetting can reduce general capabilities.
Evaluation should compare the base model and the customized model.
Training loss alone does not measure application quality.
Fine-tuning and RAG can complement each other.
Fine-tuning is usually about behavior, while RAG is often about external knowledge.
Adapters can support multiple specialized behaviors on one base model.
Fine-tuning data requires strong security and privacy controls.
Production fine-tuning requires experiment tracking and versioning.
The best model is not necessarily the largest model.
The best adaptation method is the simplest one that meets the requirements.

113. Knowledge Check

Question 1#

What is the difference between pre-training and fine-tuning?

Question 2#

What is supervised fine-tuning?

Question 3#

When is fine-tuning preferable to prompting?

Question 4#

When is RAG preferable to fine-tuning?

Question 5#

What is PEFT?

Question 6#

How does LoRA reduce the number of trainable parameters?

Question 7#

What is the purpose of LoRA rank?

Question 8#

What is QLoRA?

Question 9#

Why is quantization useful?

Question 10#

What is catastrophic forgetting?

Question 11#

Why should you evaluate the base model before fine-tuning?

Question 12#

Why is training loss not enough to evaluate a fine-tuned model?

Question 13#

Why might adapters be useful for multiple enterprise domains?

Question 14#

What kinds of problems should become regression tests after fine-tuning?

114. Next Notebook

The next notebook will move into open-source and sovereign GenAI models, model selection, local inference, quantization, and enterprise deployment:

generative_ai_open_source_sovereign_llm_models.md

It will cover:

Open-source vs open-weight models
What "sovereign AI" means
Model licensing
Data sovereignty
Model sovereignty
Major open-weight model families
Llama
Qwen
Mistral
Gemma
DeepSeek
Phi
Multimodal open models
Text, image, audio, and video capabilities
Model size selection
Dense vs Mixture-of-Experts
Quantization
GGUF
AWQ
GPTQ
Local inference
vLLM
llama.cpp
Ollama
Hugging Face deployment
GPU requirements
CPU inference
Multi-GPU deployment
Throughput and latency
Enterprise model selection
Licensing and commercial use
Sovereignty evaluation framework
Privacy and compliance
Model hosting architecture
On-premise deployment
Private cloud deployment
Air-gapped inference
Model benchmarking
Cost analysis
Practical model selection projects

Knowledge Checkpoint

Fine-Tuning, LoRA & PEFT Checkpoint

Q1.What is the core mathematical formulation of Low-Rank Adaptation (LoRA)?

AIt freezes the base weight matrix $W_0 \in \mathbb{R}^{d \times k}$ and injects trainable low-rank decomposition matrices $\Delta W = B \times A$, where $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d, k)$.

BIt removes 50% of the transformer layers at random.

CIt converts all model weights into 8-bit integers permanently.

DIt trains only the final softmax classification head.

Q2.How does QLoRA achieve fine-tuning of 70B parameter models on a single 48GB GPU?

ABy quantizing the frozen base model to 4-bit NormalFloat (NF4), using Double Quantization to compress quantization constants, and Page Optimizers to manage memory spikes.

BBy pruning 90% of model parameters.

CBy offloading all weights to CPU RAM during forward pass.

DBy training only with batch size 1.

Q3.What is the role of the LoRA scaling factor $\alpha$ (alpha)?

AIt scales the low-rank adapter contribution: $W = W_0 + \frac{\alpha}{r} (B A)$, stabilizing optimization when adjusting rank $r$.

BIt sets the GPU learning rate.

CIt controls the dropout probability.

DIt clips gradient norms.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Advanced Multimodal AI

Next Lesson

Advanced LLM Training