Advanced

32 min read

#Mixture of Experts#MoE#Sparse Routing#Scaling Laws#Top-k Gating#Advanced Architectures

31. Advanced Paradigms (Mixture of Experts & Scaling Laws)

Frontier deep learning architectures: Mixture of Experts (MoE) sparse routing, top-k expert gating, load balancing auxiliary loss, Chinchilla scaling laws, and speculative decoding.

Advanced Deep Learning: Complete Notes (Beginner to Advanced)

Introduction#

Advanced Deep Learning focuses on techniques that extend standard neural networks to handle larger models, longer contexts, multimodal data, efficient computation, self-supervised learning, and more capable decision-making systems.

The topics in this chapter connect several important ideas:

text
Large Model
    ↓
Sparse Computation ──────────────→ Mixture of Experts
    ↓
Long Context ────────────────────→ Long-Context Models
    ↓
External Knowledge ──────────────→ RAG
    ↓
Multiple Modalities ─────────────→ Multimodal RAG
    ↓
Tool / Decision Making ──────────→ Agentic Models
    ↓
Efficient Deployment ────────────→ Model Distillation
    ↓
Architecture Search ─────────────→ Neural Architecture Search
    ↓
Learning Without Manual Labels ─→ Self-Supervised Learning
    ↓
Learning Similarity / Invariance → Contrastive Learning
    ↓
Useful Representations ──────────→ Representation Learning
    ↓
Learning Through Rewards ────────→ Reinforcement Learning

1. Mixture of Experts (MoE)

1.1 What is Mixture of Experts?#

A Mixture of Experts (MoE) model contains multiple specialized neural networks called experts and a router that decides which experts should process each input.

Instead of sending every input through the entire model, an MoE model can activate only a small subset of experts.

Mathematical Formulation
                    Input
                      │
                      ▼
                   Router
                 /    |    \
                /     |     \
           Expert 1 Expert 2 Expert 3 ... Expert N
                \      |      /
                 \     |     /
                      ▼
                    Output

The important idea is:

Many parameters can exist in the model, while only a subset is activated for each input.

This is called sparse activation.

1.2 Why MoE is Useful#

A traditional dense layer activates all of its parameters for every input.

text
Dense Model

Input → Layer 1 → Layer 2 → Layer 3 → Output
          ↓         ↓         ↓
       All parameters are active

An MoE layer can instead activate selected experts.

text
MoE Model

Input
  ↓
Router
  ↓
Expert 2 + Expert 7
  ↓
Output

This can allow a model to have a very large total parameter count without requiring the full parameter set to be used for every token.

1.3 Experts#

An expert is usually a neural network module capable of processing an input representation.

In Transformer-based MoE models, experts are commonly implemented by replacing or modifying the feed-forward network (FFN) sublayer.

Conceptually:

text
Transformer Layer

Self-Attention
      ↓
MoE Feed-Forward Layer
      ↓
Next Transformer Layer

Each expert may learn different patterns during training.

The model does not necessarily assign an explicit human-defined role such as:

Mathematical Formulation
Expert 1 = mathematics
Expert 2 = programming
Expert 3 = language

Instead, specialization generally emerges through training.

1.4 Router#

The router determines which experts should receive each token.

Suppose the router produces:

text
Expert 1 → 0.05
Expert 2 → 0.70
Expert 3 → 0.10
Expert 4 → 0.15

If the model uses top-1 routing:

›Selected Expert → Expert 2

If it uses top-2 routing:

›Selected Experts → Expert 2 + Expert 4

The routing probabilities can be used to weight the expert outputs.

1.5 Sparse Activation#

Suppose an MoE model contains:

›100 experts

but only:

›2 experts

are activated for each token.

Then the model has a large total parameter pool while computation for an individual token is concentrated on a small subset.

This is the central efficiency advantage of sparse MoE architectures.

1.6 Load Balancing#

A major challenge is that the router may send too many tokens to a small number of experts.

text
Bad routing

Expert 1 → ███████████████
Expert 2 → █
Expert 3 → █
Expert 4 → █

This creates overloaded experts and underutilized experts.

A good router tries to distribute tokens more evenly:

text
Better routing

Expert 1 → █████
Expert 2 → █████
Expert 3 → █████
Expert 4 → █████

MoE systems therefore commonly use auxiliary routing/load-balancing objectives or related mechanisms.

1.7 MoE Advantages#

Large parameter capacity.
Sparse computation.
Potentially better scaling of model capacity.
Different experts can specialize.
Can reduce computation relative to activating every parameter.

1.8 MoE Challenges#

Routing complexity.
Load balancing.
Communication overhead in distributed training.
Memory requirements can still be large because all expert parameters must generally be available.
Serving can become more complicated.

1.9 Dense vs MoE Models#

Feature	Dense Model	MoE Model
Parameters used per input	Most/all	Subset
Total parameter capacity	Limited by compute budget	Can be much larger
Routing	No	Yes
Expert specialization	No explicit experts	Emerges across experts
Computation	Dense	Sparse

2. Sparse Models

2.1 What are Sparse Models?#

A sparse model is a model where only a subset of its parameters, connections, neurons, tokens, or computation paths are active for a particular input or operation.

The general idea is:

text
Dense:
████████████████████
All connections active

Sparse:
███░░░░████░░██░░░░
Only selected connections active

Sparsity can occur in different parts of a neural network.

2.2 Types of Sparsity#

Parameter Sparsity#

Some model weights are set to zero or removed.

text
Dense weights:

[0.2, 0.4, 0.7, 0.1, 0.5]

Sparse weights:

[0.2, 0,   0.7, 0,   0.5]

Activation Sparsity#

Only some neurons become active for a particular input.

Conditional Computation#

Different inputs follow different computation paths.

MoE is an important example of conditional computation.

Attention Sparsity#

A model attends to only a subset of tokens instead of every token.

This becomes especially important for long-context models.

2.3 Why Sparsity Matters#

Dense computation can become expensive as models grow.

Sparsity can potentially reduce:

computation,
memory usage,
latency,
energy consumption.

However, theoretical sparsity does not automatically translate into real hardware speedups. Efficient sparse kernels and hardware support are often required.

3. Long-Context Models

3.1 What is a Context Window?#

A model's context window is the amount of input/output token context it can process within one inference context.

text
Context

[token 1]
[token 2]
[token 3]
...
[token N]

A larger context window allows the model to consider more information at once.

3.2 Why Long Context is Important#

Consider a large document:

text
Book
 ├── Chapter 1
 ├── Chapter 2
 ├── Chapter 3
 ├── ...
 └── Chapter 20

A short-context model may need the document to be divided into smaller pieces.

A long-context model can process substantially more of the document in one context.

Applications include:

long documents,
code repositories,
legal documents,
research papers,
conversation history,
large structured inputs.

3.3 The Attention Problem#

Standard self-attention compares tokens with other tokens.

For a sequence of length N, the attention interaction matrix has approximately:

›N × N

entries.

Therefore, standard full attention has approximately:

›O(N²)

time/memory scaling with sequence length for the attention interaction itself.

If sequence length increases:

›N → 2N

the pairwise attention matrix grows from:

›N² → 4N²

This makes very long contexts expensive.

3.4 Techniques for Long Context#

Long-context systems can use techniques such as:

efficient attention mechanisms,
sparse attention,
sliding-window attention,
local/global attention,
memory mechanisms,
positional encoding improvements,
recurrent or chunked processing,
optimized inference kernels.

The exact architecture varies by model.

3.5 Long Context vs RAG#

These solve related but different problems.

text
Long Context

Large amount of information
        ↓
Put information directly into context
        ↓
Model processes it

text
RAG

Large knowledge collection
        ↓
Retrieve relevant information
        ↓
Put only relevant information into context
        ↓
Model generates answer

Long context increases how much information can be considered at once.

RAG reduces the amount of external information that needs to be placed into the context.

4. Retrieval-Augmented Generation (RAG)

4.1 What is RAG?#

Retrieval-Augmented Generation (RAG) combines:

Retrieval — finding relevant information.
Generation — using a language model to produce an answer from that information.

Basic flow:

text
User Query
    ↓
Retriever
    ↓
Relevant Documents / Chunks
    ↓
Prompt + Retrieved Context
    ↓
LLM
    ↓
Answer

4.2 Why RAG is Needed#

A model's internal parameters are not an ideal database for every piece of information.

Knowledge can also:

change over time,
be private,
be domain-specific,
be too large to memorize reliably.

RAG allows the model to access external information at inference time.

4.3 RAG Components#

1. Document Collection#

Examples:

text
PDFs
Web pages
Documentation
Databases
Company files

2. Chunking#

Large documents are divided into smaller pieces.

text
Document
   ↓
Chunk 1
Chunk 2
Chunk 3
...
Chunk N

3. Embedding#

Each chunk is converted into a vector representation.

text
Text Chunk
   ↓
Embedding Model
   ↓
[0.12, -0.41, 0.73, ...]

4. Vector Store#

The embeddings and associated metadata are stored for retrieval.

5. Retriever#

The user's query is also represented in a searchable form and compared against stored information.

6. Generator#

The retrieved context is passed to the language model.

4.4 RAG Flow#

text
                OFFLINE / INDEXING

Documents
   ↓
Chunking
   ↓
Embedding Model
   ↓
Vector Store

Then during inference:

text
                ONLINE / QUERY TIME

User Query
   ↓
Query Representation
   ↓
Retriever
   ↓
Top-K Relevant Chunks
   ↓
Prompt Construction
   ↓
LLM
   ↓
Answer

4.5 Dense Retrieval#

Dense retrieval represents text as vectors.

A similarity measure such as cosine similarity can compare query and document vectors.

text
Query embedding
       ↓
[0.2, 0.5, 0.1]
       │
       │ similarity
       ▼
Document embeddings
       ↓
Ranked results

4.6 Hybrid Retrieval#

A system can combine:

text
Dense retrieval
+
Lexical retrieval

For example:

vector similarity can capture semantic meaning,
keyword-based retrieval can capture exact terms.

Hybrid retrieval can therefore be useful when both semantic and exact matching matter.

4.7 RAG Advantages#

External knowledge can be updated without retraining the entire model.
Can use private/domain-specific information.
Retrieved sources can provide grounding.
Can reduce the need for the model to memorize every fact.

4.8 RAG Challenges#

Poor retrieval produces poor answers.
Chunking affects retrieval quality.
Too much retrieved context can distract the model.
Retrieved information can contain irrelevant or conflicting content.
Retrieval latency adds system overhead.

5. Multimodal RAG

5.1 What is Multimodal RAG?#

Multimodal RAG extends RAG beyond text.

The knowledge base can contain:

text
Text
Images
Tables
Charts
Audio
Video
Documents

The system retrieves relevant information across one or more modalities and provides it to a multimodal model.

5.2 Basic Architecture#

Architecture & Data Flow
                 Knowledge Base
              /       |       \
           Text     Images    Tables
             \         |        /
              \        |       /
             Multimodal Retrieval
                     ↓
              Relevant Evidence
                     ↓
            Multimodal Model
                     ↓
                   Answer

5.3 Example#

Suppose a user asks:

"What was the revenue shown in the chart on page 12?"

A text-only retriever might miss the important information if the number exists only inside an image or chart.

A multimodal system can:

text
PDF
 ↓
Text + page images
 ↓
Retrieve relevant page
 ↓
Analyze chart/image
 ↓
Answer

5.4 Multimodal Embeddings#

Different modalities can be represented in compatible vector spaces.

›Image → Image Encoder → Vector
Text  → Text Encoder  → Vector

If the representations are aligned:

text
Image: "cat"
        ↕
Text: "a photograph of a cat"

their embeddings can be close in the shared representation space.

This enables cross-modal retrieval.

5.5 Challenges#

Different modalities require different encoders.
Tables and charts may require specialized processing.
Images may contain text that must be extracted or understood.
Video introduces a temporal dimension.
Retrieval and ranking become more complex.

6. Agentic Models

6.1 What are Agentic Models?#

An agentic model is a model-based system designed to perform multi-step tasks by deciding what actions to take, using tools or external systems, observing results, and continuing until the task is completed or stopped.

Basic loop:

text
Goal
 ↓
Reason / Plan
 ↓
Choose Action
 ↓
Use Tool
 ↓
Observe Result
 ↓
Update State
 ↓
Choose Next Action
 ↓
...

6.2 LLM vs Agentic System#

A normal LLM interaction may look like:

›Prompt → LLM → Answer

An agentic system can look like:

text
Goal
 ↓
LLM
 ↓
Tool Call
 ↓
Tool Result
 ↓
LLM
 ↓
Another Tool Call
 ↓
Result
 ↓
Final Answer

The agent is therefore a broader system rather than simply a single neural network.

6.3 Tools#

Agents can interact with tools such as:

search systems,
databases,
APIs,
calculators,
code execution environments,
file systems,
enterprise applications.

The model decides when a tool is useful and interprets the returned result.

6.4 Planning#

A complex task can be decomposed:

text
Main Goal
   ↓
Task 1
   ↓
Task 2
   ↓
Task 3
   ↓
Final Result

Some systems explicitly generate plans; others perform planning implicitly through iterative reasoning and tool use.

6.5 Agentic RAG#

RAG and agents can be combined.

text
User Question
      ↓
Agent
      ↓
Need information?
      ↓
Retrieve
      ↓
Read evidence
      ↓
Need more information?
      ↓
Retrieve again
      ↓
Final Answer

The agent can decide which sources or retrieval operations to perform rather than relying on one fixed retrieval step.

6.6 Challenges#

Tool errors.
Incorrect action selection.
Long multi-step execution.
Unexpected tool outputs.
Security risks from tool access.
Difficulty evaluating the complete workflow.

7. Model Distillation

7.1 What is Model Distillation?#

Knowledge distillation is a technique where a smaller student model learns to reproduce useful behavior from a larger teacher model.

text
Large Teacher Model
        ↓
Teacher outputs
        ↓
Student learns
        ↓
Small Student Model

The goal is often to obtain a model that is:

smaller,
faster,
cheaper,
easier to deploy,

while retaining as much useful performance as possible.

7.2 Teacher and Student#

The teacher is typically larger or more capable.

text
Teacher
10B parameters
     ↓
Knowledge / predictions
     ↓
Student
1B parameters

The student is trained using information produced by the teacher.

7.3 Hard Labels vs Soft Targets#

Suppose a classifier predicts:

text
Cat   → 0.70
Dog   → 0.25
Horse → 0.05

The teacher's full probability distribution contains more information than only:

›Cat

These probability outputs are called soft targets.

They can communicate relationships between classes.

7.4 Distillation Loss#

A simplified distillation objective can combine ordinary supervised loss with teacher-student matching:

Mathematical Formulation
Total Loss =
    α × Student Supervised Loss
  + β × Distillation Loss

The distillation component can compare teacher and student output distributions.

Temperature scaling is often used to soften probability distributions.

text
Higher temperature
        ↓
Softer probability distribution
        ↓
More information about relative class probabilities

7.5 Distillation for Language Models#

For language models, a teacher can provide:

next-token probability distributions,
generated responses,
intermediate representations,
task-specific behavior.

A student can learn from these signals.

Distillation can therefore be used to transfer capabilities from a larger model into a smaller model.

7.6 Benefits and Limitations#

Benefits#

Smaller model.
Lower inference cost.
Lower latency.
Easier deployment.

Limitations#

Student may lose capabilities.
Distillation quality depends on the teacher.
Some capabilities are difficult to transfer.
Training still requires additional computation.

8. Neural Architecture Search (NAS)

8.1 What is NAS?#

Neural Architecture Search (NAS) is the automated process of searching for neural network architectures that perform well for a particular objective.

Instead of manually designing:

text
Layer 1 → Conv
Layer 2 → Conv
Layer 3 → Pool
Layer 4 → Dense

NAS can search over possible architectures.

8.2 NAS Components#

A NAS system generally contains:

text
Search Space
      ↓
Search Strategy
      ↓
Candidate Architecture
      ↓
Evaluation
      ↓
Performance
      ↓
Search Strategy selects another architecture

8.3 Search Space#

The search space defines what can be changed.

For example:

text
Number of layers
Filter sizes
Number of filters
Activation functions
Kernel sizes
Connections

A large search space can make the problem expensive.

8.4 Search Strategies#

Common approaches include:

Random Search#

Randomly sample architectures.

Evolutionary Search#

text
Architecture
    ↓
Mutation
    ↓
New Architecture
    ↓
Evaluation

Good architectures are retained and modified.

Reinforcement Learning#

A controller can learn to generate architectures that perform well.

Differentiable NAS#

Architecture choices can sometimes be relaxed into differentiable variables so gradient-based optimization can help search.

8.5 Why NAS is Useful#

NAS can help find architectures optimized for:

accuracy,
latency,
memory,
energy,
hardware constraints.

A production objective might be:

text
Maximize accuracy
while
latency < 20 ms
and
memory < 500 MB

9. Self-Supervised Learning

9.1 What is Self-Supervised Learning?#

Self-supervised learning trains models using supervision derived from the data itself rather than requiring manually assigned labels for every example.

The data creates a learning objective.

text
Raw Data
   ↓
Automatically create learning task
   ↓
Model prediction
   ↓
Compare with known information
   ↓
Learn representation

9.2 Example: Masked Prediction#

Consider:

›"The cat is sitting on the [MASK]."

The model predicts:

›mat

The original text provides the target.

No human needs to manually label the example as:

Mathematical Formulation
input = sentence
label = mat

9.3 Autoregressive Prediction#

Another self-supervised objective is next-token prediction.

text
Input:

The cat is

Target:

sitting

Then:

text
The cat is sitting

Target:

on

The original sequence provides the supervision.

9.4 Contrastive Self-Supervision#

Self-supervised systems can also create positive and negative examples.

text
Same image + augmented image
        ↓
Positive pair

Different images
        ↓
Negative pair

The model learns representations that capture useful similarities.

9.5 Why Self-Supervised Learning Matters#

It is especially useful when:

Mathematical Formulation
Raw data = huge
Manual labels = expensive

Instead of requiring humans to label billions of examples, the training objective can often be generated automatically.

10. Contrastive Learning

10.1 What is Contrastive Learning?#

Contrastive learning trains an encoder to bring similar examples closer in representation space and push dissimilar examples apart.

text
Positive pair
A ↔ A'
   ↓
Closer embeddings

Negative pair
A ↔ B
   ↓
Farther embeddings

10.2 Embedding Space#

Suppose:

text
Image A → [0.2, 0.5, 0.1]
Image A' → [0.21, 0.49, 0.11]

Image B → [-0.7, 0.2, 0.8]

The representation of A and A' should be similar.

The representation of B should be more distant.

10.3 Positive and Negative Pairs#

A positive pair represents related examples.

Examples:

text
Original image ↔ Augmented image
Question ↔ Relevant passage
Image ↔ Matching caption

Negative pairs represent unrelated examples.

›Image of dog ↔ Caption about a car

10.4 Contrastive Objective#

A conceptual contrastive objective is:

›Make similarity(positive pair) high
Make similarity(negative pairs) low

A common family of objectives uses a softmax over similarities.

The exact loss varies by method.

10.5 Temperature#

Contrastive objectives often use a temperature parameter.

Conceptually:

text
Similarity / Temperature
        ↓
Softmax distribution
        ↓
Contrastive loss

Temperature controls how sharply similarity differences influence the probability distribution.

10.6 Applications#

Contrastive learning is used in:

image representation learning,
text representation learning,
image-text alignment,
retrieval,
multimodal models,
self-supervised learning.

CLIP is a well-known example of contrastive learning applied to image-text representations.

11. Representation Learning

11.1 What is Representation Learning?#

Representation learning is the process of learning useful representations of raw data automatically.

Instead of manually designing every feature:

text
Raw Data
   ↓
Neural Network
   ↓
Learned Representation
   ↓
Task

11.2 Traditional Feature Engineering vs Representation Learning#

Traditional Approach#

text
Raw image
   ↓
Human-designed features
   ↓
Classifier

Deep Learning Approach#

text
Raw image
   ↓
Neural network
   ↓
Learned features
   ↓
Classifier

The network can learn increasingly abstract representations.

11.3 Hierarchical Representations#

For an image:

text
Pixels
  ↓
Edges
  ↓
Textures
  ↓
Shapes
  ↓
Objects

For language:

text
Tokens
  ↓
Word / token relationships
  ↓
Syntax
  ↓
Semantic relationships
  ↓
Higher-level meaning

These are conceptual descriptions; the exact internal representations are distributed and model-dependent.

11.4 Latent Representation#

A neural network can map high-dimensional input into a lower-dimensional or otherwise more useful latent representation.

text
Input
[large raw representation]
       ↓
Encoder
       ↓
Latent representation
       ↓
Task / Decoder

Examples include:

embeddings,
autoencoder latent spaces,
Transformer hidden states,
image features.

11.5 Good Representations#

A useful representation should capture information relevant to downstream tasks while discarding irrelevant variation when appropriate.

For example, an image representation for object recognition should ideally preserve:

text
Object identity
Shape
Important visual structure

while being less sensitive to irrelevant changes such as small translations or lighting differences, depending on the task and training objective.

12. Reinforcement Learning for Deep Learning

12.1 What is Reinforcement Learning?#

Reinforcement Learning (RL) is a learning framework where an agent interacts with an environment and learns to choose actions that maximize cumulative reward.

text
          Action
Agent ─────────────→ Environment
  ↑                       │
  │                       │
  └──── State + Reward ───┘

12.2 Core Components#

Agent#

The system making decisions.

Environment#

The world or simulation in which the agent operates.

State#

Information describing the current situation.

Action#

A decision made by the agent.

Reward#

Feedback indicating how desirable an outcome was.

Policy#

A strategy mapping states to actions.

›State → Policy → Action

12.3 Why Deep Learning is Used in RL#

Traditional RL can struggle when the state space is large.

Deep neural networks can approximate:

policies,
value functions,
action-value functions,
state representations.

This produces Deep Reinforcement Learning.

12.4 Policy Network#

A policy network can directly predict actions.

text
State
  ↓
Neural Network
  ↓
Action probabilities
  ↓
Action

For example:

text
State → [0.1, 0.7, 0.2]

Action 1 → 0.1
Action 2 → 0.7
Action 3 → 0.2

12.5 Value Function#

A value function estimates how good a state is in terms of expected future rewards.

Conceptually:

text
State
  ↓
Value Network
  ↓
Expected future return

A common definition is:

Mathematical Formulation
Vπ(s) = Eπ [ Σ γ^t r_t | s_0 = s ]

where:

Mathematical Formulation
Vπ(s) = value of state s under policy π
r_t   = reward at time t
γ     = discount factor

The discount factor controls how strongly future rewards are weighted.

12.6 Q-Function#

The action-value function estimates the expected return from taking action a in state s and then following a policy.

›Qπ(s, a)

Conceptually:

text
State + Action
      ↓
Expected future reward

12.7 Policy Gradient#

Policy-gradient methods directly optimize the policy.

Conceptually:

text
Run policy
   ↓
Collect experience
   ↓
Calculate rewards
   ↓
Estimate gradient
   ↓
Update policy
   ↓
Repeat

12.8 Actor-Critic#

Actor-Critic methods combine two components.

text
Actor
  ↓
Chooses actions

Critic
  ↓
Evaluates actions / states

The actor learns the policy.

The critic estimates value information that helps train the actor.

12.9 Reinforcement Learning for Language Models#

RL can be used after a model has learned general language behavior.

A simplified pipeline can be:

text
Pretrained Language Model
          ↓
Supervised / Instruction Fine-Tuning
          ↓
Reward Signal
          ↓
RL Optimization
          ↓
Improved Behavior

Human or AI-generated preference signals can be used to construct reward objectives.

This is one of the ideas behind approaches such as RLHF.

12.10 Exploration vs Exploitation#

An RL agent faces a trade-off.

Exploitation#

Choose actions known to produce good rewards.

Exploration#

Try actions that may reveal better strategies.

text
Too much exploitation
→ May miss better strategies

Too much exploration
→ May waste actions on poor strategies

A successful RL algorithm balances both.

13. How the Advanced Deep Learning Topics Connect

These topics are not isolated.

text
                    Deep Learning
                         │
          ┌──────────────┼──────────────┐
          │              │              │
       Scaling       Learning         Systems
          │              │              │
     ┌────┴────┐    ┌────┴────┐    ┌────┴────┐
     │         │    │         │    │         │
    MoE     Long   Self-     Contrastive   RAG
            Context Supervised Learning
     │         │    │         │       │
 Sparse      Large  Learned   Better  External
 Models      Input  Signals   Embeddings Knowledge
     │         │       │         │       │
     └─────────┴───────┴─────────┴───────┘
                         │
                  Advanced AI Systems
                         │
               ┌─────────┴─────────┐
               │                   │
          Multimodal RAG       Agentic Models
               │                   │
               └─────────┬─────────┘
                         │
                   Model Distillation
                         │
                   Efficient Models

14. Key Comparisons

14.1 MoE vs Sparse Models#

Concept	Mixture of Experts	Sparse Models
Main idea	Route input to selected experts	Activate only a subset of computation
Scope	Specific architecture/design pattern	Broad concept
Routing	Usually central	Optional
Example	Sparse expert layers	Sparse weights, sparse attention, MoE

MoE is therefore one important form of sparse/conditional computation.

14.2 Long Context vs RAG#

Feature	Long Context	RAG
Main idea	Process more tokens directly	Retrieve relevant external information
Information source	Context supplied to model	External knowledge store
Main challenge	Compute/context management	Retrieval quality
Best for	Long conversations/documents	Large or changing knowledge bases

They can also be combined:

text
Large Knowledge Base
        ↓
RAG retrieves relevant documents
        ↓
Long-context model processes more retrieved evidence
        ↓
Answer

14.3 Self-Supervised vs Contrastive Learning#

Feature	Self-Supervised Learning	Contrastive Learning
Main idea	Create supervision from data	Compare positive and negative examples
Requires manual labels	Usually not	Usually not
Learning objective	Many possible objectives	Similarity/dissimilarity objective
Relationship	Broad learning paradigm	Important self-supervised technique

Contrastive learning can therefore be self-supervised, but not all self-supervised learning is contrastive.

14.4 Representation Learning vs Embeddings#

Concept	Representation Learning	Embeddings
Meaning	Learning useful representations	A numerical representation, often a vector
Scope	Broad concept	Specific representation form
Example	CNN learning image features	Text embedding vector

An embedding can be the output of a representation-learning process.

14.5 Distillation vs Fine-Tuning#

Feature	Distillation	Fine-Tuning
Main purpose	Transfer behavior/knowledge	Adapt a model to data/task
Teacher model	Usually central	Not required
Student model	Central	Not necessarily
Typical benefit	Smaller/faster model	Task/domain adaptation

15. Practical End-to-End Example

Consider building an advanced document assistant.

Architecture & Data Flow
                    Documents
                       ↓
                Self-Supervised /
                 Pretrained Models
                       ↓
                 Representation
                    Learning
                       ↓
                Multimodal Index
                 /           \
              Text          Images
                \             /
                 \           /
              Multimodal RAG
                    ↓
             Retrieved Evidence
                    ↓
              Long-Context LLM
                    ↓
                Agentic Loop
             /       |       \
         Search    RAG      Tools
             \       |       /
                    ↓
               Final Answer

A smaller production model could then be produced through:

text
Large Teacher
     ↓
Knowledge Distillation
     ↓
Smaller Student
     ↓
Deployment

This illustrates how the concepts can work together rather than being separate technologies.

16. Summary

Topic	Core Idea
Mixture of Experts	Activate selected experts instead of all expert parameters
Sparse Models	Use only a subset of parameters/computation
Long-Context Models	Process substantially larger token contexts
RAG	Retrieve external information before generation
Multimodal RAG	Retrieve information across multiple modalities
Agentic Models	Perform multi-step actions using tools and feedback
Model Distillation	Transfer useful behavior from a larger teacher to a smaller student
Neural Architecture Search	Automatically search for effective architectures
Self-Supervised Learning	Learn using supervision derived from the data itself
Contrastive Learning	Pull related representations together and separate unrelated ones
Representation Learning	Automatically learn useful data representations
Reinforcement Learning	Learn actions through rewards and interaction

17. Quick Recap

text
MoE
→ Many experts, selectively activated.

Sparse Models
→ Only part of the computation is active.

Long-Context Models
→ Handle much larger token sequences.

RAG
→ Retrieve external knowledge before generation.

Multimodal RAG
→ Retrieve and use text, images, tables, and other modalities.

Agentic Models
→ Plan, act, observe, and repeat using tools.

Distillation
→ Transfer useful behavior from teacher to student.

NAS
→ Automatically search for neural architectures.

Self-Supervised Learning
→ Create training supervision from the data itself.

Contrastive Learning
→ Learn by comparing similar and dissimilar examples.

Representation Learning
→ Automatically learn useful features/representations.

Reinforcement Learning
→ Learn actions through rewards.

18. Final Mental Model

The easiest way to remember Advanced Deep Learning is:

text
                    ADVANCED DEEP LEARNING

        ┌────────────── Scaling ──────────────┐
        │                                      │
       MoE                              Sparse Models
        │                                      │
        └───────────────┬──────────────────────┘
                        ↓
                 Efficient Scaling
                        ↓
                 Long-Context Models
                        ↓
              External Knowledge (RAG)
                        ↓
                Multimodal RAG
                        ↓
                 Agentic Systems
                        ↓
              Advanced AI Applications


Learning Better
      │
      ├── Self-Supervised Learning
      │
      ├── Contrastive Learning
      │
      └── Representation Learning


Making Models Smaller
      │
      └── Knowledge Distillation


Finding Better Architectures
      │
      └── Neural Architecture Search


Learning Through Interaction
      │
      └── Reinforcement Learning

The central progression is:

text
Build larger models
      ↓
Make computation efficient
      ↓
Give models more context
      ↓
Connect them to external knowledge
      ↓
Support multiple modalities
      ↓
Allow them to use tools and perform multi-step tasks
      ↓
Learn better representations
      ↓
Compress capable models for deployment
      ↓
Optimize architectures and behavior

Knowledge Checkpoint

31. Advanced Paradigms & MoE Checkpoint

Q1.What is the core principle of Mixture of Experts (MoE) architectures (e.g. Mixtral 8x7B, DeepSeek-V2/V3)?

AReplacing dense feedforward (FFN) layers with multiple parallel expert sub-networks, activating only a sparse subset (e.g. top-2 of 8) per token via a learnable routing gate.

BTraining 8 completely separate models and averaging their final outputs at the end of the year.

CUsing 8 different programming languages in the training pipeline.

DRemoving all attention layers and keeping only linear layers.

Q2.Why is an Auxiliary Load Balancing Loss required during Mixture of Experts (MoE) training?

ATo prevent Router Collapse (where the gating router routes all tokens to only 1-2 favorite experts while starving remaining experts of gradients).

BTo balance electric power consumption across GPU racks.

CTo equalize dataset batch sizes.

DTo force all expert weights to be identical.

Q3.What key empirical finding did Hoffmann et al. (Chinchilla Scaling Laws) discover regarding compute-optimal LLM training?

AFor compute-optimal training, model parameter count and training dataset token count should be scaled in equal proportion (1:1), showing previous models (like GPT-3) were undertrained relative to their size.

BModels should always maximize parameter count while minimizing training tokens.

CCompute budget has zero effect on test loss.

DTransformers cannot scale beyond 1 billion parameters.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

30. Multimodal Deep Learning & Vision-Language Models