Advanced

15 min read

#generative ai#Guide

Knowledge Distillation & Model Compression for Generative AI

Comprehensive guide on Knowledge Distillation & Model Compression for Generative AI.

Knowledge Distillation & Model Compression for Generative AI

Large language models can be extremely capable, but capability often comes with substantial:

GPU memory requirements
inference latency
serving cost
energy consumption
deployment complexity

A practical AI system therefore often needs more than the strongest possible model.

It may need a model that is:

smaller
faster
cheaper
easier to deploy
suitable for edge hardware
suitable for high concurrency
easier to operate privately or on-premises

Knowledge distillation and model compression are techniques for moving in that direction.

The central idea is:

Use the behavior or information contained in a larger teacher model to build a smaller student model while preserving as much useful capability as possible.

This notebook builds on the teacher-student synthetic-data concepts from the previous notebook and goes deeper into:

knowledge distillation
response distillation
logit distillation
sequence-level distillation
instruction distillation
preference distillation
hidden-state and feature distillation
task-specific distillation
quantization
pruning
sparsity
low-rank methods
architecture-aware compression
speculative decoding
efficient serving
compression-aware evaluation
educational AI examples
production deployment

Learning Objectives

By the end of this notebook, you should be able to:

Explain knowledge distillation and the teacher-student paradigm.
Understand hard-label and soft-target distillation.
Explain temperature scaling and why it is useful.
Distinguish logit, response, feature, and preference distillation.
Build a conceptual distillation loss.
Understand sequence-level distillation for language models.
Distinguish distillation from ordinary fine-tuning.
Explain quantization and common quantization levels.
Understand pruning and sparsity.
Explain low-rank compression at a high level.
Compare model compression techniques and their trade-offs.
Design a compression pipeline for LLMs.
Evaluate capability loss, latency, memory, and cost after compression.
Build smaller educational AI models from larger teachers.
Understand why compression can fail and how to debug it.

1. Why Compress AI Models?

Suppose a teacher model has:

›100B parameters

and provides excellent quality.

Deploying it may require:

text
many GPUs
high memory
large serving cost
complex infrastructure

But perhaps the application does not need all of that capability.

A smaller model might provide:

›7B parameters

with:

text
lower memory
lower latency
lower cost
higher concurrency

The goal is not simply to make the model smaller.

The goal is:

text
Maximum useful capability
 /
Minimum deployment cost

This is an engineering optimization problem.

2. The Teacher-Student Paradigm

The basic architecture is:

Architecture & Data Flow
 TEACHER
 Large / capable
 |
 |
 knowledge / outputs
 |
 v
 STUDENT
 Small / efficient
 |
 v
 Evaluate

The teacher may provide:

labels
probability distributions
generated responses
demonstrations
critiques
preferences
intermediate representations

The student learns to reproduce useful behavior.

3. Distillation vs Fine-Tuning

These are related but different.

Fine-tuning#

Fine-tuning usually adapts a model to a dataset:

Architecture & Data Flow
Base model
 +
Task dataset
 |
 v
Fine-tuned model

4. Model Compression Taxonomy

Model compression is broader than knowledge distillation.

Major approaches include:

Architecture & Data Flow
Model Compression
|
+-- Knowledge Distillation
| +-- logit distillation
| +-- response distillation
| +-- feature distillation
|
+-- Quantization
| +-- INT8
| +-- INT4
| +-- mixed precision
|
+-- Pruning
| +-- unstructured
| +-- structured
|
+-- Low-Rank Methods
| +-- low-rank factorization
| +-- adapters
|
+-- Architecture Changes
| +-- smaller hidden size
| +-- fewer layers
| +-- smaller vocabulary
|
+-- Efficient Inference
 +-- KV cache optimization
 +-- speculative decoding

These methods can be combined.

5. Hard Targets vs Soft Targets

Suppose a classification teacher predicts:

text
cat: 0.80
dog: 0.15
fox: 0.05

A hard target might only store:

›cat

The soft distribution contains more information.

It tells the student:

text
cat is strongly preferred
dog is somewhat plausible
fox is less plausible

This additional information is often called dark knowledge.

6. Temperature Scaling

Distillation commonly uses a temperature parameter.

The softmax function becomes:

[ p_i = \frac{\exp(z_i/T)} {\sum_j \exp(z_j/T)} ]

where:

(z_i) is a logit
(T) is temperature

When:

Mathematical Formulation
T = 1

the normal distribution is used.

When:

›T > 1

the distribution becomes softer.

Example:

Mathematical Formulation
Low temperature:

A = 0.95
B = 0.04
C = 0.01

Higher temperature:

A = 0.70
B = 0.20
C = 0.10

The exact values depend on the logits.

7. Why Temperature Helps

A very confident teacher may hide useful relative information.

For example:

Mathematical Formulation
Question:
Which algorithm is most appropriate?

Teacher:
A = 0.99
B = 0.01
C = 0.00

The student learns mostly:

›A is correct

A softened distribution can reveal:

text
A is best
B is somewhat plausible
C is unlikely

This can provide richer supervision.

8. Distillation Loss

A classic distillation objective combines hard-label loss and soft-target loss.

Conceptually:

[ L = \alpha L_{hard} + (1-\alpha)L_{soft} ]

where:

(L_{hard}) measures agreement with true labels
(L_{soft}) measures agreement with teacher probabilities
(\alpha) controls the mixture

The soft loss is often based on KL divergence.

[ D_{KL}(P||Q)

\sum_i P_i \log \frac{P_i}{Q_i} ]

where:

(P) is the teacher distribution
(Q) is the student distribution

9. A PyTorch Distillation Pattern

A conceptual implementation:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn.functional as F


def distillation_loss(
 student_logits,
 teacher_logits,
 labels,
 temperature=2.0,
 alpha=0.5,
):
 hard_loss = F.cross_entropy(
 student_logits,
 labels
 )

 teacher_probs = F.softmax(
 teacher_logits / temperature,
 dim=-1
 )

 student_log_probs = F.log_softmax(
 student_logits / temperature,
 dim=-1
 )

 soft_loss = F.kl_div(
 student_log_probs,
 teacher_probs,
 reduction="batchmean"
 ) * (temperature ** 2)

 return (
 alpha * hard_loss
 + (1 - alpha) * soft_loss
 )

This is a general conceptual pattern. Real language-model training requires additional handling for sequence dimensions, padding, masking, and memory constraints.

10. Response Distillation

For large language models, direct access to teacher logits may not always be practical.

A common alternative is response distillation.

Architecture & Data Flow
Prompt
 |
 v
Teacher
 |
 v
High-quality response
 |
 v
Student training

Example:

text
Prompt:
Explain cross-validation to a beginner.

Teacher:
Cross-validation is a method...

The student is trained on the teacher's response.

This is closely related to synthetic instruction data.

11. Response Distillation Pipeline

Architecture & Data Flow
Seed / domain data
 |
 v
Teacher model
 |
 v
Generate responses
 |
 v
Quality filters
 |
 v
Distillation dataset
 |
 v
Student model
 |
 v
Evaluation

This approach can be highly practical because it does not require storing teacher logits.

12. Logit Distillation

Logit distillation uses teacher predictions before or around the softmax layer.

Conceptually:

Architecture & Data Flow
Prompt
 |
 +--> Teacher --> logits
 |
 +--> Student --> logits
 |
 v
 match teacher

Advantages:

rich probability information
direct teacher supervision
useful for classification and some LM settings

Disadvantages:

storing logits can be expensive
teacher and student vocabularies must be compatible for direct token-level matching
infrastructure becomes more complex

13. Sequence-Level Distillation

Language generation produces sequences.

Instead of matching every teacher probability, we can ask the teacher to generate target sequences.

Architecture & Data Flow
Prompt
 |
 v
Teacher
 |
 v
Generated sequence
 |
 v
Student

For example:

›Teacher:
"Cross-validation evaluates a model by..."

The student learns to reproduce useful teacher-generated sequences.

This can substantially simplify the pipeline.

14. Distillation for Instruction Following

Suppose a teacher model has strong instruction-following ability.

We can create:

Architecture & Data Flow
Instruction
 |
 v
Teacher response
 |
 v
Student training

Generate examples covering:

explanation
summarization
extraction
classification
transformation
reasoning-oriented tasks
coding
structured output
tool usage

The student can then specialize in the desired capabilities.

15. Capability-Focused Distillation

Do not necessarily distill everything.

Suppose your production application needs:

text
SQL generation
document summarization
customer support

You may build a teacher dataset specifically for these tasks.

Architecture & Data Flow
Teacher
 |
 +--> SQL
 +--> Summarization
 +--> Support
 |
 v
Student

This can produce a smaller model that is extremely effective for its target workload.

16. Domain Distillation

A general teacher can be used to create domain-specific supervision.

Example:

Architecture & Data Flow
General Teacher
 |
 v
Healthcare-style documents
 |
 v
Domain Student

The domain itself must be handled according to applicable privacy, regulatory, and safety requirements.

The teacher should not be assumed to be authoritative simply because it is larger.

17. Preference Distillation

Suppose a teacher can rank responses.

Architecture & Data Flow
Prompt
 |
 +--> Response A
 +--> Response B
 +--> Response C
 |
 v
Teacher ranking
 |
 v
Preference dataset
 |
 v
Student

The student can learn:

helpfulness
formatting
style
instruction adherence
safety preferences

This connects distillation with alignment and post-training.

18. Critique-Based Distillation

A teacher can produce both:

text
answer
critique
revision

Example:

Architecture & Data Flow
Prompt
 |
 v
Initial answer
 |
 v
Teacher critique
 |
 v
Improved answer
 |
 v
Student training

This can create higher-quality supervision than simply collecting the first teacher response.

19. Feature Distillation

Some distillation methods match intermediate representations.

Architecture & Data Flow
Teacher Layer 8
 |
 v
Teacher feature
 |
 +----> projection
 |
 v
Student Layer 4
 |
 v
Student feature

The loss encourages the student to learn representations similar to the teacher.

Challenges include:

teacher/student architecture differences
different hidden dimensions
alignment between layers
increased training complexity

20. Cross-Architecture Distillation

The teacher and student do not always need identical architectures.

For example:

text
Teacher:
large transformer

Student:
smaller transformer

Or more broadly:

Architecture & Data Flow
Large multimodal model
 |
 v
Text-only student

when the desired student capability is only textual.

Distillation therefore provides a mechanism for transferring selected capabilities across architectures.

21. Distilling Multimodal Teachers

A multimodal teacher can generate text supervision from:

text
image
audio
video
document

Then a student can learn a narrower capability.

Example:

Architecture & Data Flow
Image
 |
 v
Multimodal Teacher
 |
 +--> caption
 +--> OCR interpretation
 +--> question
 +--> answer
 |
 v
Text Student

This is useful when the production workload does not require the full multimodal teacher.

22. Distillation and Synthetic Data

These two techniques are closely related.

text
Synthetic Data:
Teacher creates useful examples.

Distillation:
Student learns from teacher-derived information.

A combined pipeline is:

Architecture & Data Flow
Teacher
 |
 +--> generate examples
 |
 v
Quality control
 |
 v
Student training
 |
 v
Compressed model

The previous notebook focused primarily on generating high-quality datasets. This notebook focuses on using those datasets and teacher information to compress capability into smaller systems.

23. Quantization

Quantization reduces numerical precision.

For example:

Architecture & Data Flow
FP32
 |
 v
FP16 / BF16
 |
 v
INT8
 |
 v
INT4

Lower precision can reduce:

model memory
memory bandwidth
serving cost

But aggressive quantization can reduce quality.

24. Parameter Memory Estimation

A rough estimate:

Mathematical Formulation
Memory ≈ number_of_parameters × bytes_per_parameter

For a model with:

›7 billion parameters

using FP16:

Mathematical Formulation
7B × 2 bytes
≈ 14 GB

This is only a rough weight-memory estimate.

Actual serving memory also includes:

KV cache
activations
runtime overhead
CUDA/framework allocations
temporary buffers

25. Quantization Example

Suppose:

text
FP16:
2 bytes / parameter

INT8:
1 byte / parameter

INT4:
0.5 bytes / parameter

For 7B parameters, the rough weight sizes are:

Architecture & Data Flow
FP16 -> ~14 GB
INT8 -> ~7 GB
INT4 -> ~3.5 GB

Real implementations have metadata and packing overhead, so these numbers are approximations.

26. Post-Training Quantization

A model can often be quantized after training.

Architecture & Data Flow
Trained model
 |
 v
Calibration
 |
 v
Quantization
 |
 v
Compressed model
 |
 v
Evaluation

Calibration data should represent the real workload reasonably well.

27. Quantization-Aware Training

Another approach incorporates quantization effects during training.

Architecture & Data Flow
Training
 |
 +--> simulate quantization
 |
 v
Model learns to tolerate precision loss

This can preserve quality better in some scenarios but increases training complexity.

28. Quantization Trade-Off

Think of a three-way trade-off:

Architecture & Data Flow
 Quality
 ^
 |
 |
 |
 +------------>
 Memory / Cost

Lower precision often means:

text
lower memory
lower cost
potentially lower quality

The correct choice depends on the workload.

29. Calibration

Quantization often requires representative data.

Example:


🐍 PythonInteractive WebAssembly
calibration_samples = [
 "Explain machine learning.",
 "Write a SQL query.",
 "Summarize this document.",
]

A stronger calibration set should represent:

real prompts
typical sequence lengths
domain vocabulary
structured outputs
important edge cases

30. Pruning

Pruning removes parameters or connections that contribute less to the desired behavior.

Conceptually:

Architecture & Data Flow
Dense model
1111111111
1111111111
1111111111

 |
 v

Pruned model
1100101001
0100100010
1001100001

The zeros represent removed or inactive connections.

31. Unstructured vs Structured Pruning

Unstructured pruning#

Individual weights are removed.

Mathematical Formulation
W =

1.2 0.0 0.4
0.0 0.7 0.0

This can produce high sparsity but may not always translate directly into faster hardware execution.

Structured pruning#

Entire structures are removed:

neurons
channels
attention heads
layers
blocks

Structured pruning is often easier to exploit with conventional hardware.

32. Sparsity

A model with 50% sparsity has roughly half its target weights set to zero.

text
Dense:
100% weights

Sparse:
50% weights

But:

50% fewer non-zero weights does not automatically mean 2× faster inference.

Speed depends on:

hardware
kernels
memory access
sparsity pattern
serving framework

33. Low-Rank Compression

A large matrix can sometimes be approximated by smaller matrices.

Instead of:

[ W \in \mathbb{R}^{m \times n} ]

we approximate:

[ W \approx AB ]

where:

[ A \in \mathbb{R}^{m \times r} ]

and:

[ B \in \mathbb{R}^{r \times n} ]

with:

[ r \ll \min(m,n) ]

This can reduce the number of parameters.

34. Low-Rank Intuition

Imagine:

Architecture & Data Flow
Original matrix

[m x n]

 |
 v

Factorization

[m x r] [r x n]

where r is much smaller

The approximation works well when the original matrix contains redundancy that can be represented with fewer dimensions.

35. Architecture-Level Compression

Instead of compressing only weights, redesign the model.

Possible changes:

fewer layers
smaller hidden dimension
fewer attention heads
smaller intermediate dimension
smaller vocabulary
grouped-query attention
efficient attention mechanisms

Example:

Architecture & Data Flow
Large Teacher
48 layers
4096 hidden

 |
 v

Smaller Student
24 layers
2048 hidden

The student then requires substantially fewer parameters.

36. Layer Dropping

One simple compression strategy is removing selected layers.

text
Teacher:

L1
L2
L3
L4
L5
L6
L7
L8

Student:

L1
L2
L4
L6
L8

Layer dropping can be combined with subsequent fine-tuning or distillation.

The layers should not be assumed to be equally removable.

37. Vocabulary Compression

Token vocabularies can contribute to embedding and output-layer size.

For specialized applications, vocabulary design may sometimes be optimized.

However, changing the tokenizer can create significant compatibility and training challenges.

Therefore vocabulary compression is more specialized than ordinary quantization.

38. Knowledge Distillation + Quantization

These methods can be combined.

Architecture & Data Flow
Large Teacher
 |
 v
Distill
 |
 v
Small Student
 |
 v
Quantize
 |
 v
Efficient Student

This can be powerful because:

›Distillation -> recover capability
Quantization -> reduce memory

But each compression stage needs independent evaluation.

39. Knowledge Distillation + Pruning

Another pipeline:

Architecture & Data Flow
Teacher
 |
 v
Student
 |
 v
Pruning
 |
 v
Fine-tune / Distill again
 |
 v
Compressed student

After pruning, additional training may recover some lost capability.

40. A Multi-Stage Compression Pipeline

A production workflow may look like:

Architecture & Data Flow
 Teacher
 |
 v
 Capability analysis
 |
 v
 Distillation
 |
 v
 Student model
 |
 +--------+--------+
 | |
 v v
 Quantization Pruning
 | |
 +--------+--------+
 |
 v
 Calibration
 |
 v
 Evaluation
 |
 +-----+-----+
 | |
 Pass Fail
 | |
 v v
 Deploy Revisit

Do not compress everything at once.

Incremental evaluation makes failures easier to diagnose.

41. Evaluating Compression

A compressed model should be evaluated on at least four dimensions:

text
1. Capability
2. Quality
3. Efficiency
4. Reliability

Capability examples:

accuracy
coding success
reasoning benchmarks
instruction following

Efficiency:

model memory
TTFT
tokens/sec
throughput
GPU count

Reliability:

failure rate
timeout rate
output-format compliance
safety behavior

42. Compression Ratio

A simple metric:

[ Compression\ Ratio = \frac{Original\ Model\ Size} {Compressed\ Model\ Size} ]

Example:

Mathematical Formulation
Original = 14 GB
Compressed = 3.5 GB

Compression ratio = 4x

But size reduction alone is not enough.

A 4× smaller model that loses 30% of required quality may be unacceptable.

43. Quality Retention

A useful metric:

[ Quality\ Retention = \frac{Compressed\ Quality} {Teacher\ or\ Baseline\ Quality} ]

Suppose:

Mathematical Formulation
Baseline = 90
Compressed = 87

Then:

Mathematical Formulation
Retention = 87 / 90
 ≈ 96.7%

Use the metric carefully because different evaluation metrics may not be linearly comparable.

44. Pareto Thinking

Compression should be viewed as a Pareto problem.

You want good combinations of:

text
Quality
Latency
Memory
Cost
Throughput

A model is attractive if no alternative is:

text
better quality
AND
lower cost
AND
lower latency

simultaneously.

45. Example Model Comparison

Suppose:

Model	Quality	Memory	Latency
Teacher	92	80 GB	900 ms
Student FP16	89	14 GB	250 ms
Student INT8	88	8 GB	190 ms
Student INT4	86	4.5 GB	160 ms

A product decision depends on requirements.

For a strict quality workload:

›Student FP16

may be preferable.

For high concurrency:

›Student INT4

may be attractive if the quality remains acceptable.

46. Distillation Dataset Design

A distillation dataset should reflect the target workload.

For an educational assistant:

text
Explanation 25%
Question solving 20%
Feedback 15%
Code 10%
Summarization 10%
Misconceptions 10%
Structured output 10%

These numbers are illustrative.

The distribution should be derived from product requirements and evaluation results.

47. Distilling a Specialized Educational Model

Imagine a large teacher model understands an entire ML curriculum.

The production student only needs to:

explain concepts
answer exercises
provide hints
detect misconceptions
generate quizzes

Pipeline:

Architecture & Data Flow
Large Teacher
 |
 v
Curriculum specification
 |
 v
Generate targeted examples
 |
 v
Quality control
 |
 v
Distill into small student
 |
 v
Educational evaluation

48. Student Model Evaluation for Education

Measure more than generic language quality.

Evaluate:

factual correctness
pedagogical clarity
age/level appropriateness
misconception handling
hint quality
curriculum alignment
answer consistency
hallucination
safety

A smaller model may be acceptable if it performs extremely well on these target metrics.

49. Distilling Tool Use

A teacher can demonstrate tool-use behavior.

Example:

Architecture & Data Flow
User question
 |
 v
Teacher
 |
 +--> tool call
 |
 +--> tool result
 |
 +--> final answer
 |
 v
Student dataset

The student can learn structured tool-calling patterns.

However, tool execution must remain protected by runtime permissions and validation.

50. Distilling Structured Outputs

Suppose the teacher generates:

json
{
 "topic": "linear_regression",
 "difficulty": "intermediate",
 "question_type": "conceptual"
}

A student can be trained to reliably produce the same schema.

Evaluation should validate the schema directly.


🐍 PythonInteractive WebAssembly
def valid_output(obj):
 return (
 isinstance(obj, dict)
 and "topic" in obj
 and "difficulty" in obj
 and "question_type" in obj
 )

51. Distillation for Edge AI

Small models are especially useful on:

laptops
mobile devices
embedded systems
private on-premises servers
low-cost cloud instances

A possible pipeline:

Architecture & Data Flow
Large cloud teacher
 |
 v
Distillation
 |
 v
Small model
 |
 v
Quantization
 |
 v
Edge deployment

This can reduce dependency on cloud inference.

52. Distillation for Sovereign AI

In environments requiring strong data sovereignty:

Architecture & Data Flow
External / large teacher
 |
 v
Approved training pipeline
 |
 v
Sovereign student
 |
 v
On-prem deployment

Once the student is trained and validated, inference can happen within the controlled environment.

Careful governance is still required around:

training-data provenance
licensing
model licenses
privacy
supply chain
evaluation
deployment isolation

53. Quantization Formats and Tooling

Common model-compression ecosystems include:

PyTorch quantization tooling
bitsandbytes
GPTQ-style quantization
AWQ-style quantization
GGUF-based local inference
hardware-specific quantization libraries

The best format depends on:

model architecture
hardware
inference engine
target precision
workload

Do not choose a format only because it is popular.

54. Inference-Aware Compression

A compression technique is valuable only if the deployment stack can exploit it.

For example:

Architecture & Data Flow
Sparse weights
 |
 v
Hardware
 |
 +--> efficient sparse kernel?
 |
 yes/no

If the runtime cannot exploit the sparse structure, memory or latency improvements may be smaller than expected.

55. KV Cache and Compression

For autoregressive LLMs, weights are not the only memory consumer.

During generation:

Architecture & Data Flow
Prompt
 |
 v
KV Cache
 |
 v
Generated tokens

Long contexts and many concurrent requests can make KV cache memory significant.

Therefore:

Model weight compression and KV-cache optimization solve different memory problems.

Both may matter in production.

56. Speculative Decoding

Speculative decoding can improve generation latency using a smaller draft model.

Architecture & Data Flow
 Draft model
 |
 v
 Candidate tokens
 |
 v
 Large model
 |
 v
 Verify candidates

The large model remains responsible for verification.

This is not the same as distilling the large model into the draft model, but the two techniques can work together.

57. Distillation + Speculative Decoding

A useful architecture:

Architecture & Data Flow
Teacher / target model
 |
 v
Distillation
 |
 v
Small draft model
 |
 +----------------+
 |
 v
 Speculative decoding
 |
 v
 Faster generation

The draft model can be trained specifically to predict tokens that the target model is likely to accept.

58. Compression Failure Modes

Failure 1: Student copies teacher errors#

If teacher outputs are wrong, the student can inherit them.

Failure 2: Student becomes too narrow#

Over-specialized distillation data can cause capability loss outside the target domain.

Failure 3: Over-quantization#

Aggressive precision reduction can damage quality.

Failure 4: Pruning removes useful structures#

Important layers or attention patterns may be damaged.

Failure 5: Dataset bias#

If the distillation dataset overrepresents one capability, the student may overfit to it.

Failure 6: Teacher-student mismatch#

Very different architectures can make some forms of direct feature/logit distillation difficult.

Failure 7: Benchmark overfitting#

Optimizing compression against one benchmark can hide broader regressions.

59. Compression Regression Testing

Every compression candidate should pass the same evaluation suite.

Architecture & Data Flow
Baseline
 |
 v
Evaluation suite
 |
 +--> capability
 +--> safety
 +--> latency
 +--> memory
 |
 v
Compression candidate
 |
 v
Same evaluation suite
 |
 v
Compare

This creates an objective before/after measurement.

60. Evaluation Slices

Do not report only one average score.

Break results down by:

task
language
difficulty
input length
output length
domain
modality
safety category

Example:

text
Overall: 89%
Beginner: 95%
Advanced: 82%
Long context: 76%
Code: 91%
Math: 84%

Compression may affect specific slices much more than the average.

61. Distillation Monitoring

Track:

text
Teacher quality
Student quality
Quality gap
Compression ratio
Latency
Throughput
Memory
Cost
Failure rate

A useful experiment table:

Version	Params	Precision	Quality	Memory	TTFT	Cost
Teacher	70B	FP16	92	140 GB	700 ms	High
Student A	14B	FP16	89	28 GB	280 ms	Medium
Student B	14B	INT8	88	15 GB	220 ms	Lower
Student C	7B	INT4	85	5 GB	140 ms	Low

Values are illustrative.

62. Production Architecture

A compression program can be organized as:

Architecture & Data Flow
 Model Registry
 |
 v
 Teacher Selection
 |
 v
 Distillation Jobs
 |
 +-----------+-----------+
 | |
 v v
 Dataset Store Teacher Outputs
 | |
 +-----------+-----------+
 |
 v
 Student Training
 |
 v
 Compression Pipeline
 |
 +---------------+---------------+
 | | |
 v v v
 Quantization Pruning Architecture
 | | |
 +---------------+---------------+
 |
 v
 Evaluation
 |
 v
 Model Registry
 |
 v
 Deployment

63. Model Registry Metadata

A compressed model should record:

json
{
 "model_id": "edu-student-7b-int4",
 "base_model": "student-7b",
 "teacher_model": "teacher-v3",
 "distillation_dataset": "edu-distill-v5",
 "precision": "int4",
 "compression_methods": [
 "distillation",
 "quantization"
 ],
 "quality_score": 0.91,
 "memory_gb": 5.1
}

This makes model lineage explicit.

64. Cost Optimization

Suppose:

text
Teacher:
$10 / million generated tokens

Student:
$0.30 / million tokens

If the student preserves sufficient quality, moving high-volume traffic to the student can dramatically reduce operating cost.

A practical routing architecture:

Architecture & Data Flow
 Request
 |
 v
 Model Router
 |
 +------------+------------+
 | |
 v v
 Small Student Large Teacher
 common requests complex requests

The router can use:

task type
confidence
latency target
customer tier
complexity
sovereignty requirements

65. Cascaded Models

A model cascade can be:

Architecture & Data Flow
Request
 |
 v
Small model
 |
 +--> confident --> answer
 |
 +--> uncertain --> large model

This can reduce cost while preserving quality on difficult requests.

Confidence estimation should be validated; a language model's raw confidence is not automatically calibrated.

66. Project 1: Basic Knowledge Distillation

Build a teacher-student experiment using a classification dataset.

Tasks:

train or load a teacher
create a smaller student
implement hard-label training
implement soft-target distillation
compare performance
vary temperature

Measure:

text
Teacher accuracy
Student hard-label accuracy
Student distilled accuracy

67. Project 2: LLM Response Distillation

Create a small instruction dataset.

Pipeline:

Architecture & Data Flow
Prompts
 |
 v
Teacher
 |
 v
Responses
 |
 v
Quality filter
 |
 v
Student fine-tuning

Compare:

base student
fine-tuned student
distilled student

Evaluate target capabilities.

68. Project 3: Quantization Benchmark

Take one model and compare:

text
FP16
INT8
INT4

Measure:

memory
load time
TTFT
tokens/sec
task quality

Create a report showing the quality-efficiency trade-off.

69. Project 4: Distillation for Educational AI

Create a teacher dataset containing:

text
concept explanation
exercise
hint
solution
misconception
feedback
quiz question

Train a smaller model.

Evaluate it on:

correctness
clarity
difficulty alignment
misconception handling
response latency

70. Project 5: Compression Pipeline

Build:

Architecture & Data Flow
Teacher
 |
 v
Distillation
 |
 v
Student
 |
 v
Quantization
 |
 v
Benchmark

Produce a table containing:

text
parameter count
model size
memory
latency
throughput
quality
cost estimate

71. Project 6: Model Cascade

Deploy:

Architecture & Data Flow
7B student
 |
 +--> simple request -> student
 |
 +--> complex request -> 70B teacher

Define routing rules.

Measure:

percentage handled by student
average latency
total cost
quality
escalation rate

72. Advanced Exercise: Distillation Dataset Ablation

Train students with:

Architecture & Data Flow
Dataset A -> human data
Dataset B -> synthetic teacher data
Dataset C -> human + synthetic
Dataset D -> high-quality filtered synthetic

Compare:

capability
hallucination
diversity
generalization

This reveals how much synthetic teacher data actually contributes.

73. Advanced Exercise: Compression Stack

Create several candidates:

text
Student FP16
Student INT8
Student INT4
Student + pruning
Student + INT4 + pruning

Measure every candidate against the same evaluation suite.

Plot the quality-versus-memory trade-off.

74. Advanced Exercise: Distillation Failure Analysis

Find examples where:

Mathematical Formulation
Teacher = correct
Student = incorrect

Classify the failures:

text
factual
reasoning
formatting
long-context
instruction-following
domain-specific

Then generate targeted additional distillation examples.

75. Advanced Exercise: Distill a Multimodal Teacher

Use a multimodal teacher to generate:

›image -> question -> answer

Then train a smaller text model to answer the generated questions.

Investigate:

text
teacher grounding errors
student transfer quality
cross-modal information loss

76. Common Mistakes

Mistake 1: Assuming the largest teacher is automatically perfect#

Teachers have errors and biases.

Mistake 2: Distilling without a target capability definition#

A student can become broadly mediocre rather than narrowly excellent.

Mistake 3: Measuring only model size#

A smaller model is useful only if it meets quality and operational requirements.

Mistake 4: Quantizing without representative calibration data#

Quality can degrade unexpectedly on real workloads.

Mistake 5: Assuming sparsity equals speed#

Hardware and runtime support determine whether sparse models actually run faster.

Mistake 6: Compressing multiple dimensions simultaneously#

When quality drops, it becomes difficult to identify the cause.

Mistake 7: Evaluating only one benchmark#

Compression can cause slice-specific regressions.

Mistake 8: Ignoring serving infrastructure#

A theoretically compressed model may not produce real-world gains if the inference engine cannot exploit the format.

Mistake 9: Treating distillation as a one-time operation#

A mature system should continuously evaluate and improve the student.

77. Practical Compression Decision Framework

Ask these questions in order:

Step 1: What is the target workload?#

text
chat
coding
education
classification
RAG
agents
multimodal

Step 2: What is the quality requirement?#

›minimum acceptable score

Step 3: What is the deployment constraint?#

text
CPU
single GPU
multi-GPU
edge device
mobile
air-gapped

Step 4: What is the latency target?#

text
TTFT
tokens/sec
p95 latency

Step 5: What is the budget?#

text
cost/request
cost/1M tokens
GPU-hours

Step 6: Choose compression methods#

text
distillation
quantization
pruning
architecture reduction

78. Final Mental Model

Think of the teacher as a large, expensive expert and the student as a compact specialist.

Architecture & Data Flow
 EXPERT
 Large Teacher
 |
 +------------+------------+
 | | |
 outputs rankings features
 | | |
 +------------+------------+
 |
 v
 Distillation
 |
 v
 SMALL STUDENT
 |
 +-----------+-----------+
 | |
 v v
 Quantization Pruning
 | |
 +-----------+-----------+
 |
 v
 EFFICIENT MODEL
 |
 v
 Production system

The key insight is:

Compression is successful when the model becomes substantially cheaper or faster while retaining the capabilities that actually matter for the target workload.

Key Takeaways

Knowledge distillation transfers useful teacher behavior into a student model.
Distillation can use hard labels, soft probabilities, generated responses, preferences, or intermediate features.
Temperature controls the softness of teacher probability distributions.
Response distillation is especially practical for LLMs when teacher logits are expensive to store.
Synthetic data generation and distillation naturally work together.
Quantization reduces numerical precision to reduce memory and potentially improve efficiency.
Pruning removes parameters or structures but does not automatically guarantee proportional speedups.
Low-rank methods exploit redundancy in weight matrices.
Architecture-level compression can reduce layers, hidden dimensions, or other structural components.
Compression techniques can be combined, but each stage should be evaluated.
Model size is only one dimension; quality, latency, throughput, memory, cost, and reliability matter too.
Compression should be evaluated using representative workload slices.
Smaller specialized models can be excellent for educational AI and other focused applications.
Model cascades can combine cheap students with powerful teachers.
The best compressed model is usually not the smallest model; it is the model that meets the product requirements at the best efficiency.

Knowledge Check

Question 2#

What is the difference between hard and soft targets?

Question 3#

Why does temperature matter in distillation?

Question 5#

How is distillation different from ordinary fine-tuning?

Question 7#

Why does 50% sparsity not necessarily produce 2× faster inference?

Question 9#

Why should compression be evaluated using slices rather than only one overall benchmark?

Question 10#

How can a model cascade reduce inference cost?

Course Progression

Completed:

text
01 Generative AI & LLM Foundations
02 Transformers & LLM Architecture
03 RAG, Embeddings & Vector Databases
04 LangChain, LangGraph & Agentic AI
05 LLM Evaluation, Safety & Guardrails
06 Multimodal Generative AI
07 Fine-Tuning, LoRA, QLoRA & PEFT
08 Open-Source, Open-Weight & Sovereign LLMs
09 LLMOps & Inference Optimization
10 End-to-End Generative AI Projects
11 AI Application Security & Governance
12 Advanced RAG & Agent Architectures
13 AI Platform Architecture & Engineering
14 Distributed Inference & GPU Engineering
15 Data Engineering & Evaluation Infrastructure
16 Advanced Evaluation & Benchmarking
17 Synthetic Data & Dataset Generation
18 Knowledge Distillation & Model Compression

›19 Advanced LLM Training

The next notebook moves from compressing trained models to understanding how large language models are trained at scale: data pipelines, distributed training, objectives, parallelism, memory optimization, checkpoints, training stability, scaling behavior, and practical training architecture.

Knowledge Checkpoint

Distillation & Model Compression Checkpoint

Q1.What is Knowledge Distillation in Generative AI?

ATraining a compact student model to replicate the output probability distribution (soft logits) and feature representations of a large, high-capacity teacher model.

BDeleting 50% of the dataset rows.

CExporting weights to a CSV file.

DCompressing model weights with gzip.

Q2.What is the difference between Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT)?

APTQ quantizes weights after training without further backpropagation, while QAT models quantization error during training forward passes to recover lost accuracy.

BPTQ is for GPUs; QAT is for CPUs.

CPTQ converts weights to strings; QAT converts them to floats.

DQAT does not require training data.

Q3.What is Structured Pruning versus Unstructured Pruning in neural networks?

AStructured pruning removes entire attention heads, channels, or layers to yield immediate hardware acceleration; unstructured pruning zeroes individual scalar weights, requiring specialized sparse matrix hardware.

BStructured pruning deletes user prompts; unstructured pruning deletes outputs.

CStructured pruning applies only to decision trees.

DThere is no difference.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Post-Training & Alignment for Generative AI

Next Lesson

Small Language Models & Edge AI