Advanced

24 min read

#Multimodal#Vision-Language#CLIP#Cross-Attention#Contrastive Learning#Embedding Fusion

30. Multimodal Deep Learning & Vision-Language Models

Bridging vision and language: OpenAI CLIP contrastive pre-training, modality-specific encoders, cross-attention projection bottlenecks, and visual instruction tuning.

Multimodal Deep Learning: Complete Notes (Beginner to Advanced)

Introduction#

Multimodal Deep Learning focuses on models that learn from and combine information from multiple modalities.

A modality is a type of data, such as:

text
Text
Images
Audio
Video
Speech
Sensor data

A unimodal model may process only one type of information:

›Image → Vision Model → Prediction

A multimodal model can combine different types:

text
Image + Text
     ↓
Multimodal Model
     ↓
Joint Understanding
     ↓
Output

For example:

text
Image:
[photo of a dog]

Text:
"What is the dog doing?"

        ↓

Multimodal Model

        ↓

"The dog is running."

A high-level multimodal pipeline is:

text
Modality A ──► Encoder A ──┐
                           │
Modality B ──► Encoder B ──┼──► Fusion / Multimodal Model ──► Output
                           │
Modality C ──► Encoder C ──┘

1. Multimodal Models

1.1 What is a Multimodal Model?#

A multimodal model is a model that can process information from two or more modalities and use those representations together.

Examples:

text
Image + Text
Audio + Text
Video + Text
Image + Audio + Text

The central challenge is:

text
Different modalities
       ↓
Different representations
       ↓
Align / combine representations
       ↓
Joint understanding

1.2 Why Multimodal Learning?#

Different modalities contain complementary information.

For example:

text
Image
→ visual appearance

Text
→ language and semantic description

Combining them can provide richer information than either modality alone.

Example:

text
Image:
A person holding an umbrella

Text:
"Is it raining?"

Combined understanding:
The image provides visual evidence,
while the text specifies the question.

1.3 Modality-Specific Encoders#

A common architecture uses a separate encoder for each modality.

text
Image
  ↓
Vision Encoder
  ↓
Image Representation

Text
  ↓
Text Encoder
  ↓
Text Representation

These representations can then be:

text
Concatenated
Projected
Aligned
Cross-attended

depending on the architecture.

1.4 Multimodal Fusion#

Fusion is the process of combining information from different modalities.

Common conceptual approaches include:

Early Fusion#

Combine representations relatively early.

text
Image Features ──┐
                 ├──► Combined Representation
Text Features ───┘

Late Fusion#

Process modalities separately and combine predictions later.

text
Image → Model A → Prediction ──┐
                               ├──► Combined Prediction
Text  → Model B → Prediction ──┘

Intermediate Fusion#

Process each modality independently for some layers, then combine learned representations.

text
Image → Encoder ──┐
                   ├──► Fusion → Joint Model
Text  → Encoder ──┘

Modern multimodal Transformers often use more sophisticated forms of intermediate fusion and cross-attention.

2. Vision-Language Models

2.1 What is a Vision-Language Model?#

A Vision-Language Model (VLM) combines visual and language information.

It can process:

text
Images
+
Text

and learn relationships between them.

Typical tasks include:

Image-text retrieval
Image captioning
Visual question answering
Image understanding
Visual grounding
Document understanding

2.2 Basic VLM Architecture#

A simplified VLM can look like:

text
Image
  ↓
Vision Encoder
  ↓
Image Features
  │
  ├──────────────┐
  │              │
  ▼              ▼
Projection     Fusion
  │              │
  │         Text Representation
  │              ▲
  │              │
Text → Tokenizer → Text Encoder
                 │
                 ▼
          Multimodal Model
                 │
                 ▼
               Output

Different VLMs use different architectures. Some align image and text embeddings directly, while others feed visual representations into a language model.

2.3 Example: Visual Question Answering#

Input:

text
Image:
[image of a red car]

Question:
"What color is the car?"

The model processes:

text
Image → Visual Representation
Text  → Language Representation
                ↓
             Fusion
                ↓
             Answer
                ↓
             "Red"

3. Image Embeddings

3.1 What are Image Embeddings?#

An image embedding is a numerical vector representing an image in a learned feature space.

text
Image
  ↓
Vision Encoder
  ↓
Image Embedding

Example:

text
Image
  ↓
[0.12, -0.45, 0.87, ..., 0.21]

The vector may contain hundreds or thousands of dimensions depending on the model.

3.2 What Does an Image Embedding Represent?#

An image embedding can capture useful visual information such as:

text
Objects
Shapes
Textures
Colors
Visual patterns
Semantic concepts

The exact information depends on how the encoder was trained.

3.3 Image Embedding Example#

A vision model can transform:

›Image A

into:

›v_A

and:

›Image B

into:

›v_B

If the model's embedding space places semantically similar images near each other, then:

›similarity(v_A, v_B)

can be high.

Cosine similarity is commonly used:

Mathematical Formulation
cosine_similarity(a,b)
=
(a · b) / (||a|| ||b||)

4. Text Embeddings

4.1 What are Text Embeddings?#

A text embedding is a numerical vector representing text in a learned semantic space.

text
Text
 ↓
Text Encoder
 ↓
Text Embedding

Example:

text
"Golden retriever running"

        ↓

[0.18, -0.31, 0.92, ..., 0.14]

4.2 What Do Text Embeddings Represent?#

Depending on the model and training objective, text embeddings can capture relationships involving:

text
Meaning
Semantic similarity
Concepts
Entities
Language patterns

For example:

›"dog"

and:

›"puppy"

may have embeddings that are closer than unrelated words.

5. Multimodal Embeddings

5.1 What are Multimodal Embeddings?#

Multimodal embeddings are representations designed to place information from different modalities into a shared or compatible representation space.

For example:

text
Image
  ↓
Image Encoder
  ↓
Image Embedding
      │
      │ Shared Space
      │
Text Embedding
  ↑
Text Encoder
  ↑
Text

The goal is for semantically related image and text representations to be close.

5.2 Example#

Suppose we have:

text
Image:
[dog running in a park]

Text:
"A dog is running outside."

A multimodal embedding model may produce:

›Image → v_image
Text  → v_text

with high similarity:

›similarity(v_image, v_text) → high

For unrelated text:

›"A computer motherboard"

the similarity should ideally be lower.

5.3 Shared Embedding Space#

A shared embedding space allows cross-modal operations.

For example:

text
Text Query
   ↓
Text Embedding
   ↓
Compare against image embeddings
   ↓
Retrieve matching images

This enables:

›Text → Image Search
Image → Text Search

6. Vision Transformers (ViT)

6.1 What is Vision Transformer?#

Vision Transformer (ViT) is a Transformer-based architecture for image processing.

Instead of processing an image directly with convolution operations, ViT divides the image into fixed-size patches and treats those patches as a sequence of tokens.

Conceptually:

text
Image
  ↓
Split into Patches
  ↓
Patch Embeddings
  ↓
Transformer
  ↓
Image Representation

6.2 Image Patches#

Suppose an image has dimensions:

›224 × 224

and patch size:

›16 × 16

Number of patches:

Mathematical Formulation
(224 / 16) × (224 / 16)
=
14 × 14
=
196 patches

Each patch is converted into a vector.

Therefore:

text
Image
 ↓
196 Patch Tokens
 ↓
Transformer

6.3 Patch Embedding#

Each image patch can be flattened:

›16 × 16 × 3

for an RGB image.

That gives:

›768 values

The flattened patch is projected into the Transformer embedding dimension.

Conceptually:

text
Patch
 ↓
Flatten
 ↓
Linear Projection
 ↓
Patch Embedding

6.4 Positional Information#

Transformers do not inherently know the spatial arrangement of patches.

Therefore, positional information is added.

text
Patch Embedding
      +
Position Information
      ↓
Transformer

This allows the model to distinguish:

›Patch at top-left

from:

›Patch at bottom-right

6.5 ViT Architecture#

A simplified ViT:

text
Image
  ↓
Patch Extraction
  ↓
Patch Embeddings
  +
Positional Embeddings
  ↓
Transformer Encoder
  ↓
Transformer Encoder
  ↓
...
  ↓
Image Representation
  ↓
Classification Head

Some ViT implementations use a special class token:

›[CLS] + Patch1 + Patch2 + ... + PatchN

The representation associated with the class token can be used for classification.

6.6 ViT vs CNN#

Feature	CNN	ViT
Basic unit	Pixels / local regions	Image patches as tokens
Main operation	Convolution	Self-attention
Spatial inductive bias	Strong	Weaker by default
Global relationships	Built through layers	Self-attention can model them directly
Architecture	Convolutional	Transformer-based

ViTs often benefit substantially from large-scale pretraining.

7. CLIP

7.1 What is CLIP?#

CLIP (Contrastive Language-Image Pre-training) is a multimodal model designed to learn aligned representations of images and text.

It uses:

text
Image Encoder
+
Text Encoder

and trains them so that matching image-text pairs have similar representations.

7.2 CLIP Architecture#

text
             Image
               │
               ▼
         Image Encoder
               │
               ▼
        Image Embedding
               │
               │
               │ Similarity
               │
               ▼
         Shared Space
               ▲
               │
               │
        Text Embedding
               ▲
               │
         Text Encoder
               ▲
               │
              Text

The image and text encoders produce representations that can be compared.

7.3 Contrastive Learning#

Suppose a batch contains:

text
Image 1 ↔ Text 1
Image 2 ↔ Text 2
Image 3 ↔ Text 3

Correct pairs should have high similarity:

text
Image 1 ↔ Text 1  → High
Image 2 ↔ Text 2  → High
Image 3 ↔ Text 3  → High

Incorrect pairs should have lower similarity:

›Image 1 ↔ Text 2 → Low
Image 1 ↔ Text 3 → Low

The training objective encourages matching pairs to be closer in the shared representation space.

7.4 CLIP Similarity Matrix#

Suppose:

›3 images
3 text descriptions

The model can calculate a similarity matrix:

text
             Text 1  Text 2  Text 3

Image 1       high    low     low
Image 2       low     high    low
Image 3       low     low     high

Training encourages the diagonal entries to have high similarity.

7.5 Zero-Shot Classification#

One important capability of CLIP-style models is zero-shot classification.

Suppose the image is:

›[photo of a dog]

Create text descriptions:

text
"a photo of a dog"
"a photo of a cat"
"a photo of a car"

The model computes:

›Image ↔ Text similarities

Example:

text
dog → 0.91
cat → 0.12
car → 0.04

The highest similarity corresponds to:

›dog

No task-specific classifier needs to be trained for those labels.

8. Multimodal Transformers

8.1 What are Multimodal Transformers?#

Multimodal Transformers extend Transformer-based processing to multiple modalities.

They can combine:

text
Text
+
Image
+
Audio
+
Video

depending on the model.

8.2 Basic Architecture#

A multimodal Transformer may use modality-specific encoders:

text
Image → Vision Encoder ──┐
                         │
Text  → Text Encoder ────┼──► Multimodal Transformer
                         │
Audio → Audio Encoder ───┘
                                  │
                                  ▼
                               Output

Another architecture can convert multiple modalities into token-like representations and process them jointly.

8.3 Cross-Attention#

One important mechanism for multimodal interaction is cross-attention.

Suppose:

›Text representation → Query
Image representation → Key + Value

Then:

text
Text Queries
      ↓
Cross-Attention
      ↑
Image Keys / Values

This allows text representations to selectively attend to visual information.

8.4 Example#

Question:

›"What color is the car?"

The text representation generates queries.

The image representation provides keys and values.

text
Text
 ↓
Queries
 ↓
Cross-Attention ← Image Features
 ↓
Relevant Visual Information
 ↓
Answer

The model can focus on image regions or representations relevant to the question.

8.5 Joint Multimodal Transformer#

A more integrated architecture can represent modalities as a combined sequence:

text
[Text Tokens]
+
[Image Tokens]
+
[Other Modality Tokens]

Then:

text
Combined Multimodal Sequence
          ↓
Transformer
          ↓
Joint Representation
          ↓
Task Output

The exact architecture depends on the model.

9. Multimodal Fusion

A multimodal system must determine how information from different modalities interacts.

text
Image Representation
        │
        ├──────────┐
        │          │
        ▼          ▼
      Fusion    Cross-Attention
        │          │
        └────┬─────┘
             ▼
       Joint Representation

Common strategies include:

text
Concatenation
Projection
Cross-Attention
Shared Embedding Space
Joint Transformer

10. Image-Text Retrieval

Multimodal embeddings enable image-text retrieval.

Suppose a database contains:

text
Image 1
Image 2
Image 3
...
Image N

A user enters:

›"dog playing in snow"

Pipeline:

text
Text Query
    ↓
Text Encoder
    ↓
Text Embedding
    ↓
Similarity Search
    ↓
Image Embeddings
    ↓
Rank Results
    ↓
Most Similar Images

This is a major application of shared multimodal embedding spaces.

11. Image Captioning

Image captioning generates natural-language descriptions of images.

Conceptually:

text
Image
 ↓
Vision Encoder
 ↓
Visual Representation
 ↓
Language Model
 ↓
Text Generation
 ↓
Caption

Example:

text
Image:
[dog running through grass]

Output:
"A dog is running through a grassy field."

A vision-language generation model may use cross-attention or another mechanism to condition language generation on visual information.

12. Visual Question Answering

Visual Question Answering (VQA) combines:

text
Image
+
Question

to generate an answer.

text
Image ──► Vision Encoder ──┐
                           ├──► Multimodal Model ──► Answer
Question ─► Text Encoder ──┘

Example:

text
Image:
[person riding a bicycle]

Question:
"What is the person riding?"

Answer:
"A bicycle."

13. Multimodal Embeddings vs Multimodal Transformers

These concepts are related but different.

Multimodal Embeddings#

Focus on:

text
Representations
      ↓
Shared / aligned space
      ↓
Similarity / retrieval

Example:

›Image ↔ Text similarity

Multimodal Transformers#

Focus on:

text
Multiple modalities
      ↓
Deep interaction
      ↓
Joint reasoning / generation

Example:

text
Image + Question
      ↓
Transformer
      ↓
Answer

A multimodal Transformer can also produce embeddings, so these categories can overlap.

14. CLIP vs Generative Vision-Language Models

Feature	CLIP-style Model	Generative VLM
Main goal	Align image and text representations	Understand and/or generate language conditioned on visual input
Image encoder	Yes	Usually yes
Text component	Text encoder	Often language model
Shared embedding space	Central to training	May or may not be central
Image-text retrieval	Excellent use case	Possible
Text generation	Not the primary purpose	Core capability
Visual question answering	Not the primary design	Common capability

15. ViT vs CLIP vs Multimodal Transformer

Model / Concept	Main Role
ViT	Transformer-based vision encoder
CLIP	Align image and text representations
Multimodal Transformer	Jointly process/interact across multiple modalities

A common relationship can be:

text
Image
 ↓
ViT-like Vision Encoder
 ↓
Image Representation
 ↓
CLIP Alignment / Multimodal Transformer
 ↓
Multimodal Task

Not every system uses exactly this architecture.

16. Simple Conceptual PyTorch Example

A simplified multimodal architecture can be represented as:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn


class SimpleMultimodalModel(nn.Module):

    def __init__(
        self,
        image_dim,
        text_dim,
        hidden_dim
    ):
        super().__init__()

        self.image_projection = nn.Linear(
            image_dim,
            hidden_dim
        )

        self.text_projection = nn.Linear(
            text_dim,
            hidden_dim
        )

        self.classifier = nn.Linear(
            hidden_dim * 2,
            3
        )

    def forward(
        self,
        image_embedding,
        text_embedding
    ):
        image_features = self.image_projection(
            image_embedding
        )

        text_features = self.text_projection(
            text_embedding
        )

        combined = torch.cat(
            [
                image_features,
                text_features
            ],
            dim=-1
        )

        return self.classifier(combined)

The flow is:

text
Image Embedding
      ↓
Projection ───────┐
                  │
                  ├──► Concatenate ──► Classifier
                  │
Text Embedding    │
      ↓           │
Projection ───────┘

This is a simplified example for understanding multimodal fusion, not a complete production VLM.

17. Important Distinctions

Image Embedding vs Text Embedding#

text
Image Embedding
→ Numerical representation of visual information

Text Embedding
→ Numerical representation of textual information

Multimodal Embedding#

text
Image + Text
      ↓
Compatible / Shared Representation Space

The key goal is cross-modal semantic alignment.

ViT#

text
Image
 ↓
Patches
 ↓
Transformer
 ↓
Visual Representation

ViT itself is primarily a vision architecture, not necessarily a multimodal model.

CLIP#

text
Image → Image Encoder ──┐
                        ├──► Shared Embedding Space
Text  → Text Encoder ───┘

Its core idea is contrastive alignment between image and text.

Multimodal Transformer#

text
Multiple Modalities
       ↓
Transformer Interaction
       ↓
Joint Representation
       ↓
Understanding / Generation

18. Summary Table

Concept	Core Idea
Multimodal Model	Processes and combines multiple data modalities
Vision-Language Model	Combines visual and language information
Image Embedding	Vector representation of an image
Text Embedding	Vector representation of text
Multimodal Embedding	Aligns representations across modalities
ViT	Uses Transformer architecture over image patches
CLIP	Learns aligned image-text representations through contrastive learning
Multimodal Transformer	Uses Transformer mechanisms to interact across modalities
Fusion	Combines information from different modalities
Cross-Attention	Allows one modality to attend to another
Image-Text Retrieval	Finds matching images from text or vice versa
Image Captioning	Generates text describing an image
VQA	Answers questions using visual and language information

19. Quick Recap

text
                 MULTIMODAL DEEP LEARNING
                           │
            ┌──────────────┼──────────────┐
            │              │              │
          Image           Text          Audio
            │              │              │
            ▼              ▼              ▼
       Vision Encoder  Text Encoder  Audio Encoder
            │              │              │
            ▼              ▼              ▼
       Image Embedding Text Embedding Audio Embedding
            │              │              │
            └──────────────┼──────────────┘
                           ▼
                    Fusion / Alignment
                           │
                           ▼
                 Multimodal Representation
                           │
                           ▼
              Understanding / Retrieval /
                 Classification / Generation

Core mental model#

text
ViT
→ Understand images using Transformer architecture

Image Embedding
→ Convert an image into a vector

Text Embedding
→ Convert text into a vector

Multimodal Embedding
→ Put different modalities into a compatible representation space

CLIP
→ Learn image-text alignment using contrastive learning

Vision-Language Model
→ Combine vision and language for tasks such as VQA and captioning

Multimodal Transformer
→ Allow multiple modalities to interact using Transformer mechanisms

The central idea is:

text
Different Modalities
        ↓
Modality-Specific Representations
        ↓
Alignment / Fusion
        ↓
Joint Multimodal Understanding
        ↓
Prediction / Retrieval / Generation

Knowledge Checkpoint

30. Multimodal Deep Learning Checkpoint

Q1.How is OpenAI's CLIP (Contrastive Language-Image Pre-training) trained across image-text pairs?

AUsing a symmetric cross-entropy contrastive loss that maximizes cosine similarity between matching (image_i, text_i) embeddings while minimizing similarity with non-matching pairs in a batch.

BBy training an image generator to draw text descriptions from scratch.

CBy classifying images into 1,000 ImageNet categories.

DBy translating text into audio speech waveforms.

Q2.In modern Vision-Language Models (e.g. LLaVA), how are image visual tokens integrated into the LLM autoregressive decoder?

AA vision encoder (CLIP ViT) extracts image patch features, a linear or MLP projector maps them into the LLM embedding dimension, and they are prepended as visual tokens in the input context.

BImages are converted to ASCII art text strings.

CThe vision encoder directly predicts text tokens without an LLM.

DThe LLM is replaced with a convolutional network.

Q3.What is Late Fusion in multimodal model architectures?

AProcessing each modality (audio, vision, text) independently through specialized deep encoders until final high-level representation layers, combining them only before the decision output.

BCombining raw image pixels with audio waveforms at layer 1.

CDelaying model inference by 5 seconds.

DTraining the vision model after the text model has converged.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

29. Large Language Models (LLM) Pre-training & Inference

Next Lesson

31. Advanced Paradigms (Mixture of Experts & Scaling Laws)