Advanced

28 min read

#Transformers#Self-Attention#Multi-Head Attention#Positional Encoding#Encoder-Decoder#Architecture

18. Transformers Architecture & Multi-Head Self-Attention

Attention Is All You Need: Scaled dot-product self-attention, Multi-Head Attention, sinusoidal and rotary positional encodings, feed-forward sublayers, and layer normalization.

Transformers: Complete Notes (Beginner to Advanced)

Introduction#

A Transformer is a neural network architecture designed to process sequences using attention mechanisms instead of relying on recurrence as its primary method of sequence interaction.

The Transformer architecture was introduced in the paper "Attention Is All You Need" and became the foundation of many modern language and sequence models.

The central idea is:

text
Sequence
   ↓
Token Representations
   ↓
Positional Information
   ↓
Attention
   ↓
Feed-Forward Processing
   ↓
Repeated Transformer Layers
   ↓
Contextual Representations

A Transformer can be built using:

›Encoder
Decoder

or only one of them, depending on the model and task.

The original Transformer is an encoder-decoder architecture.

A simplified view is:

Architecture & Data Flow
                 Transformer
                      |
             +--------+--------+
             |                 |
             v                 v
          Encoder           Decoder
             |                 |
             |                 |
             +--------+--------+
                      |
                      v
                   Output

The main concepts covered in this topic are:

text
Transformer Architecture
Transformer Encoder
Transformer Decoder
Encoder-Decoder Architecture
Self-Attention
Multi-Head Attention
Positional Encoding
Positional Embeddings
Feed-Forward Network
Residual Connections
Layer Normalization
Causal Attention
Encoder Attention
Decoder Attention

1. Transformer Architecture

The Transformer is composed of repeated layers containing attention and feed-forward operations.

The original Transformer contains:

text
Encoder Stack
+
Decoder Stack

A simplified architecture is:

Architecture & Data Flow
                 INPUT SEQUENCE
                       |
                       v
                Token Embeddings
                       |
                       v
             Positional Encoding
                       |
                       v
              +----------------+
              | Encoder Layer  |
              +----------------+
                       |
                       v
              +----------------+
              | Encoder Layer  |
              +----------------+
                       |
                       v
                Encoder Output
                       |
                       |----------------------+
                       |                      |
                       v                      |
              +----------------+              |
              | Decoder Layer  |<-------------+
              +----------------+
                       |
                       v
              +----------------+
              | Decoder Layer  |
              +----------------+
                       |
                       v
                Linear Projection
                       |
                       v
                    Softmax
                       |
                       v
                 Output Tokens

Transformer Encoder Layer#

A typical encoder layer contains:

text
Input
  ↓
Multi-Head Self-Attention
  ↓
Add + Layer Normalization
  ↓
Feed-Forward Network
  ↓
Add + Layer Normalization
  ↓
Output

Transformer Decoder Layer#

A typical original Transformer decoder layer contains:

text
Input
  ↓
Masked Multi-Head Self-Attention
  ↓
Add + Layer Normalization
  ↓
Cross-Attention
  ↓
Add + Layer Normalization
  ↓
Feed-Forward Network
  ↓
Add + Layer Normalization
  ↓
Output

The exact normalization ordering can differ across Transformer implementations. The sequence above describes the common original post-normalization formulation.

2. Transformer Encoder

The Transformer Encoder converts an input sequence into contextual representations.

For example:

›"The cat sat"

is first converted into token representations.

The encoder then allows the tokens to interact through self-attention.

Encoder Flow#

text
Input Tokens
     ↓
Token Embeddings
     ↓
Positional Information
     ↓
Self-Attention
     ↓
Feed-Forward Network
     ↓
Encoder Output

This process is repeated through multiple encoder layers.

Encoder Layer#

Architecture & Data Flow
          Input
            |
            v
     Multi-Head Self-Attention
            |
            v
      Residual Connection
            |
            v
      Layer Normalization
            |
            v
     Feed-Forward Network
            |
            v
      Residual Connection
            |
            v
      Layer Normalization
            |
            v
          Output

What Does the Encoder Produce?#

The encoder produces contextual representations:

text
h₁
h₂
h₃
...
hₙ

Each representation contains information influenced by other positions that the attention mechanism was allowed to access.

For a non-causal encoder, a token can generally attend to tokens on both sides.

Conceptually:

›Token 1 ↔ Token 2 ↔ Token 3 ↔ Token 4

The encoder itself does not necessarily produce the final task output. Its representations can be passed to a decoder or another task-specific head.

3. Transformer Decoder

The Transformer Decoder generates or transforms an output sequence using decoder self-attention and, in an encoder-decoder Transformer, information from the encoder.

A typical original Transformer decoder layer contains three major components:

text
Masked Self-Attention
        ↓
Cross-Attention
        ↓
Feed-Forward Network

Decoder Flow#

text
Target Tokens
     ↓
Token Embeddings
     ↓
Positional Information
     ↓
Masked Self-Attention
     ↓
Cross-Attention
     ↑
Encoder Output
     ↓
Feed-Forward Network
     ↓
Output Representation
     ↓
Linear Layer
     ↓
Softmax
     ↓
Next Token Prediction

Why Masked Self-Attention?#

During autoregressive generation, a token should not use information from future target positions.

For:

›A B C D

the attention pattern is:

text
A → A

B → A B

C → A B C

D → A B C D

This is implemented using causal attention.

Decoder's Two Attention Mechanisms#

The original encoder-decoder Transformer decoder has:

›1. Decoder Self-Attention
2. Encoder-Decoder Cross-Attention

The first uses the decoder's own sequence.

The second uses:

text
Queries → Decoder
Keys    → Encoder
Values  → Encoder

4. Encoder-Decoder Architecture

An Encoder-Decoder Architecture contains two main networks:

›Encoder
Decoder

The encoder processes the input sequence.

The decoder generates the output sequence.

Complete Flow#

Architecture & Data Flow
Source Sequence
      |
      v
   Encoder
      |
      v
Encoder Representations
      |
      |-----------------------+
      |                       |
      |                       v
      |                Cross-Attention
      |                       ^
      |                       |
      v                       |
   Decoder ------------------+
      |
      v
Output Sequence

Example: Translation#

Suppose the source is:

›"I like cats"

The encoder creates contextual representations:

text
Encoder
   ↓
Representations

The decoder then generates:

›"J'aime les chats"

During generation, the decoder can use cross-attention to retrieve relevant information from the encoder.

Why Separate Encoder and Decoder?#

The two components have different responsibilities:

text
Encoder
→ understand / represent the source sequence

Decoder
→ generate the target sequence

This design is particularly useful when input and output sequences differ.

Examples include:

text
Machine Translation
Text Summarization
Sequence Transformation

5. Self-Attention

Self-Attention allows tokens within the same sequence to interact with one another.

Given input representations:

›X

the model creates:

Mathematical Formulation
Q = XW_Q
K = XW_K
V = XW_V

and calculates:

Mathematical Formulation
Attention(Q,K,V)
=
softmax(QKᵀ / √dₖ)V

Encoder Self-Attention#

In an encoder, self-attention is generally non-causal.

For:

›x₁ x₂ x₃ x₄

the attention pattern can conceptually be:

›x₁ ↔ x₂ ↔ x₃ ↔ x₄

Every position can use information from other positions, subject to any additional masks.

Decoder Self-Attention#

In an autoregressive decoder, self-attention is causal:

text
x₁ → x₁

x₂ → x₁ x₂

x₃ → x₁ x₂ x₃

x₄ → x₁ x₂ x₃ x₄

This prevents future-token information leakage.

Why Self-Attention?#

Self-attention allows the model to capture relationships between distant tokens directly.

For example:

›"The animal that the children saw was tired."

The representation of one word can attend to relevant words elsewhere in the sequence.

6. Multi-Head Attention

Multi-Head Attention performs several attention operations in parallel.

Instead of one attention transformation:

text
Q, K, V
   ↓
Attention
   ↓
Output

multiple heads are created:

Architecture & Data Flow
Q, K, V
   |
   +----> Head 1
   |
   +----> Head 2
   |
   +----> Head 3
   |
   +----> ...
   |
   +----> Head h
            |
            v
        Concatenate
            |
            v
     Output Projection
            |
            v
          Output

For each head:

Mathematical Formulation
headᵢ =
Attention(QW_Q⁽ⁱ⁾, KW_K⁽ⁱ⁾, VW_V⁽ⁱ⁾)

Then:

Mathematical Formulation
MultiHead(Q,K,V)
=
Concat(head₁, ..., head_h)W_O

Why Multiple Heads?#

Different heads can learn different patterns of relationships.

For example, heads may learn useful patterns involving:

text
Local relationships
Long-range dependencies
Syntactic relationships
Semantic relationships

These are possible learned patterns, not fixed roles assigned by the architecture.

Encoder Use#

The encoder uses:

›Multi-Head Self-Attention

Decoder Use#

The original Transformer decoder uses:

text
Masked Multi-Head Self-Attention
+
Multi-Head Cross-Attention

7. Positional Encoding

Transformers do not inherently process tokens sequentially like RNNs.

Self-attention itself does not provide information about the order of tokens.

Therefore, the model needs positional information.

Positional Encoding is a method of injecting position information into token representations.

The original Transformer uses deterministic sinusoidal positional encodings.

For position pos and dimension i:

Mathematical Formulation
PE(pos, 2i)
=
sin(pos / 10000^(2i/d_model))

PE(pos, 2i+1)
=
cos(pos / 10000^(2i/d_model))

Where:

Mathematical Formulation
pos       = token position
i         = dimension index
d_model   = model embedding dimension

Why Positional Encoding?#

Consider:

›"dog bites man"

and:

›"man bites dog"

The same words are present, but their order is different.

The model needs positional information to distinguish the arrangements.

Adding Positional Encoding#

The original Transformer adds positional encoding to token embeddings:

Mathematical Formulation
Input Representation
=
Token Embedding
+
Positional Encoding

Conceptually:

text
Token Embedding
      +
Position Information
      ↓
Transformer Input

8. Positional Embeddings

Positional Embeddings are learned or otherwise parameterized representations that encode token positions.

Instead of using fixed sinusoidal functions, a model can learn a vector for each position.

Conceptually:

text
Position 1 → learned vector p₁
Position 2 → learned vector p₂
Position 3 → learned vector p₃
...

The input can then be represented as:

›Token Embedding + Positional Embedding

Positional Encoding vs Positional Embeddings#

These terms are sometimes used loosely, but an important distinction is:

text
Positional Encoding
→ positional information generated by a fixed mathematical scheme
  such as sinusoidal functions.

Positional Embedding
→ learned positional vectors.

The original Transformer used sinusoidal positional encodings rather than learned positional embeddings.

Modern Transformer architectures can use different positional-information methods, so these should not be treated as one universal implementation.

Example#

Suppose:

text
Token Embedding:
E(token)

Position Embedding:
P(position)

Then:

Mathematical Formulation
Representation
=
E(token) + P(position)

9. Feed-Forward Network

The Feed-Forward Network (FFN) is a neural network applied independently to each sequence position after the attention operation.

A standard Transformer FFN is commonly:

Mathematical Formulation
FFN(x)
=
W₂ σ(W₁x + b₁) + b₂

where σ is typically a nonlinear activation.

In the original Transformer, the activation was ReLU.

Important Property#

The same FFN parameters are applied independently at every token position.

For:

›x₁ x₂ x₃ x₄

the operation is conceptually:

text
x₁ → FFN → y₁
x₂ → FFN → y₂
x₃ → FFN → y₃
x₄ → FFN → y₄

The FFN does not itself mix information between positions.

Attention performs the major cross-position interaction.

Encoder Layer#

text
Input
 ↓
Self-Attention
 ↓
FFN
 ↓
Output

Why FFN?#

Attention determines:

›Which information should interact?

The feed-forward network then performs nonlinear transformation of each resulting representation.

A useful conceptual separation is:

text
Attention
→ communication between positions

FFN
→ computation/transformation within each position

10. Residual Connections

A Residual Connection, or skip connection, adds a layer's input to its output.

The basic equation is:

Mathematical Formulation
y = x + F(x)

In a Transformer, an attention or feed-forward sublayer is combined with the sublayer input through a residual connection.

Conceptually:

Architecture & Data Flow
          x
          |
          +------------------+
          |                  |
          v                  |
      Sublayer F(x)          |
          |                  |
          +--------+---------+
                   |
                   v
                  Add
                   |
                   v
                 Output

Transformer Example#

For an attention sublayer:

Mathematical Formulation
Attention Output = F(x)

Residual Output = x + F(x)

In the original Transformer formulation, normalization is applied around these residual blocks according to the post-normalization structure.

Why Residual Connections?#

Residual connections help:

Improve gradient flow
Make optimization of deep networks easier
Allow a layer to learn a residual modification instead of needing to completely replace its input

Without a residual path:

›x → F(x)

With a residual path:

Mathematical Formulation
x → F(x)
 \    /
  \  /
   Add
    ↓
x + F(x)

Residual connections are especially important when many Transformer layers are stacked.

11. Layer Normalization

Layer Normalization normalizes activations across the feature dimensions of an individual example.

For an activation vector:

Mathematical Formulation
x = [x₁, x₂, ..., x_d]

the mean is:

Mathematical Formulation
μ = (1/d) Σ xᵢ

and variance is:

Mathematical Formulation
σ² = (1/d) Σ (xᵢ - μ)²

The normalized value is:

Mathematical Formulation
x̂ᵢ = (xᵢ - μ) / √(σ² + ε)

A learnable scale and shift are then applied:

Mathematical Formulation
yᵢ = γx̂ᵢ + β

Where:

text
γ = learnable scale
β = learnable shift
ε = small constant for numerical stability

Why Layer Normalization?#

Layer normalization helps stabilize the activations used throughout the network.

It can make optimization more stable and predictable.

Transformer Usage#

The original Transformer uses layer normalization around the attention and feed-forward sublayers.

Conceptually:

text
Input
 ↓
Sublayer
 ↓
Add Residual
 ↓
LayerNorm

Modern Transformer implementations also commonly use pre-normalization:

text
Input
 ↓
LayerNorm
 ↓
Sublayer
 ↓
Add Residual

Therefore, when discussing Transformer normalization, it is important to distinguish:

text
Post-LN
→ Sublayer → Add → LayerNorm

Pre-LN
→ LayerNorm → Sublayer → Add

12. Causal Attention

Causal Attention prevents a position from attending to future positions.

It is essential for autoregressive decoding.

For a sequence:

›x₁ x₂ x₃ x₄

the allowed attention pattern is:

text
        Keys
        1  2  3  4

Q1      ✓  ✗  ✗  ✗
Q2      ✓  ✓  ✗  ✗
Q3      ✓  ✓  ✓  ✗
Q4      ✓  ✓  ✓  ✓

This is a lower-triangular pattern.

Masking#

Before softmax, future positions are masked.

Conceptually:

text
Allowed position → original score

Future position → -∞

Then:

›softmax(masked_scores)

assigns approximately zero probability to forbidden positions.

Why Causal Attention?#

Suppose the model is predicting:

›"The cat is ___"

It should not be able to inspect the actual future token during training.

Therefore:

text
Current position
→ can see past + current
→ cannot see future

This preserves the autoregressive property.

13. Encoder Attention

Encoder Attention refers primarily to the self-attention mechanism used inside Transformer encoder layers.

The encoder receives:

›X

and creates:

Mathematical Formulation
Q = XW_Q
K = XW_K
V = XW_V

Then:

Mathematical Formulation
Attention(Q,K,V)
=
softmax(QKᵀ / √dₖ)V

Encoder Attention Is Normally Non-Causal#

For an encoder sequence:

›x₁ x₂ x₃ x₄

a token can generally attend to:

text
Past
+
Current
+
Future

subject to any padding or task-specific masks.

Conceptually:

›x₁ ↔ x₂ ↔ x₃ ↔ x₄

Why?#

The encoder is typically processing the complete source/input sequence.

For tasks such as:

text
Understanding a sentence
Text classification
Sequence representation

there is no need to hide future source tokens.

14. Decoder Attention

The original Transformer decoder uses two attention mechanisms.

›1. Decoder Self-Attention
2. Encoder-Decoder Cross-Attention

14.1 Decoder Self-Attention#

This attention is causal.

text
Target tokens
     ↓
Masked Self-Attention
     ↓
Decoder representation

For:

›y₁ y₂ y₃ y₄

the pattern is:

text
y₁ → y₁

y₂ → y₁ y₂

y₃ → y₁ y₂ y₃

y₄ → y₁ y₂ y₃ y₄

14.2 Encoder-Decoder Cross-Attention#

The decoder then attends to the encoder output.

The sources are:

text
Q → Decoder
K → Encoder
V → Encoder

Conceptually:

Architecture & Data Flow
Decoder State
     |
     v
    Query
     |
     +----------------+
                      |
Encoder Output → Keys + Values
                      |
                      v
               Cross-Attention
                      |
                      v
             Decoder Representation

This allows the decoder to retrieve relevant information from the input sequence.

Complete Decoder Layer#

Architecture & Data Flow
Target Input
     |
     v
Masked Self-Attention
     |
     v
Residual + LayerNorm
     |
     v
Cross-Attention
     ↑
Encoder Output
     |
     v
Residual + LayerNorm
     |
     v
Feed-Forward Network
     |
     v
Residual + LayerNorm
     |
     v
Decoder Output

15. Encoder vs Decoder

The encoder and decoder have different attention structures.

Component	Encoder	Original Decoder
Self-Attention	Yes	Yes
Self-Attention Mask	Usually non-causal	Causal
Cross-Attention	No	Yes
Feed-Forward Network	Yes	Yes
Residual Connections	Yes	Yes
Layer Normalization	Yes	Yes
Main Role	Encode source/input	Generate target/output

Simple Mental Model#

text
ENCODER
"What does the input mean?"

DECODER
"What should I generate next,
using what I have generated and
what the encoder understood?"

16. Complete Transformer Data Flow

For an encoder-decoder Transformer:

Architecture & Data Flow
                SOURCE SEQUENCE
                       |
                       v
               Token Embeddings
                       |
                       v
             Positional Information
                       |
                       v
              +----------------+
              | Encoder Layer  |
              |                |
              | Self-Attention |
              |       ↓        |
              |     FFN        |
              +----------------+
                       |
                       v
                 Encoder Output
                       |
                       |
                       +----------------------+
                                              |
                                              v
TARGET SEQUENCE                        Cross-Attention
       |                                      ^
       v                                      |
Token Embeddings                              |
       |                                      |
       v                                      |
Positional Information                        |
       |                                      |
       v                                      |
+-------------------+                         |
| Decoder Layer     |                         |
|                   |                         |
| Causal Self-Attn  |                         |
|        ↓          |                         |
| Cross-Attention --+-------------------------+
|        ↓          |
|       FFN         |
+-------------------+
       |
       v
Linear Projection
       |
       v
Softmax
       |
       v
Output Token

17. Transformer Layer-by-Layer View

A Transformer is usually deep because multiple layers are stacked.

Encoder#

text
Input
 ↓
Encoder Layer 1
 ↓
Encoder Layer 2
 ↓
Encoder Layer 3
 ↓
...
 ↓
Encoder Layer N
 ↓
Encoder Output

Decoder#

text
Target Input
 ↓
Decoder Layer 1
 ↓
Decoder Layer 2
 ↓
Decoder Layer 3
 ↓
...
 ↓
Decoder Layer N
 ↓
Decoder Output

Each layer repeatedly performs:

text
Attention
+
Feed-Forward Transformation
+
Residual Connections
+
Layer Normalization

18. Attention Types Inside the Original Transformer

There are three important attention operations.

Encoder Self-Attention#

text
Q ← Encoder
K ← Encoder
V ← Encoder

Purpose:

›Allow source tokens to interact with one another.

Decoder Self-Attention#

text
Q ← Decoder
K ← Decoder
V ← Decoder

with causal masking.

Purpose:

›Allow generated/previous target tokens to interact
without seeing future target tokens.

Encoder-Decoder Cross-Attention#

text
Q ← Decoder
K ← Encoder
V ← Encoder

Purpose:

›Allow the decoder to retrieve relevant information
from the encoded source sequence.

19. Positional Information vs Attention

These concepts solve different problems.

Attention#

Answers:

›Which other tokens are relevant to this token?

Positional Information#

Answers:

›Where is this token located in the sequence?

Therefore:

text
Token Embedding
      +
Positional Information
      ↓
Transformer Representation
      ↓
Attention

Without positional information, standard self-attention by itself does not inherently encode sequence order.

20. Residual + Normalization + Sublayer

A Transformer layer can be understood as a repeated block.

Original Post-Normalization Style#

Conceptually:

Architecture & Data Flow
Input
  |
  +-----------------------+
  |                       |
  v                       |
Sublayer                  |
  |                       |
  +----------> Add <------+
                 |
                 v
            LayerNorm
                 |
                 v
               Output

The same pattern is then used for another sublayer.

For example:

text
x
 ↓
Self-Attention
 ↓
Add + LayerNorm
 ↓
Feed-Forward
 ↓
Add + LayerNorm

Pre-Normalization Style#

Modern implementations often use:

text
x
 ↓
LayerNorm
 ↓
Sublayer
 ↓
Add Residual
 ↓
Output

The exact placement matters for optimization behavior, especially in deep Transformer stacks.

21. Transformer Architecture Summary

Component	Purpose
Transformer	Attention-based sequence architecture
Encoder	Builds contextual representations of input
Decoder	Generates/produces output representations
Encoder-Decoder	Maps one sequence to another
Self-Attention	Allows positions within the same source to interact
Multi-Head Attention	Performs multiple attention patterns in parallel
Positional Encoding	Adds position information using a fixed scheme
Positional Embeddings	Learned vectors representing positions
Feed-Forward Network	Applies nonlinear transformation independently per position
Residual Connection	Adds input directly to sublayer output
Layer Normalization	Normalizes feature activations
Causal Attention	Prevents attention to future positions
Encoder Attention	Usually non-causal self-attention
Decoder Attention	Causal self-attention plus cross-attention in original encoder-decoder Transformer

22. Quick Recap

text
Transformer
→ Attention-based architecture for sequence processing.

Encoder
→ Processes the source sequence and creates contextual representations.

Decoder
→ Processes target-side representations and generates output.

Encoder-Decoder
→ Encoder understands the source; decoder generates the target.

Self-Attention
→ Q, K, V come from the same sequence.

Multi-Head Attention
→ Multiple attention heads operate in parallel.

Positional Encoding
→ Adds position information using a fixed mathematical scheme.

Positional Embeddings
→ Learned vectors representing positions.

Feed-Forward Network
→ Nonlinear transformation applied independently to each position.

Residual Connection
→ Adds the sublayer input to its output.

Layer Normalization
→ Normalizes activations across feature dimensions.

Causal Attention
→ Prevents a position from seeing future positions.

Encoder Attention
→ Usually non-causal self-attention.

Decoder Attention
→ Causal self-attention + encoder-decoder cross-attention.

Final Mental Model

Architecture & Data Flow
                         TRANSFORMER
                              |
                 +------------+------------+
                 |                         |
                 v                         v
              ENCODER                   DECODER
                 |                         |
                 |                    Causal Self-Attn
                 |                         |
                 |                         v
                 |                  Cross-Attention
                 |                    ↑
                 |                    |
                 +--------------------+
                              |
                              v
                         Feed-Forward
                              |
                              v
                     Residual + LayerNorm
                              |
                              v
                           Output

The most important flow to remember is:

Architecture & Data Flow
                ENCODER
                   |
                   v
        Token + Position Information
                   |
                   v
          Self-Attention
                   |
                   v
          Feed-Forward Network
                   |
                   v
             Encoder Output
                   |
                   |
                   +----------------------+
                                          |
                                          v
                DECODER            Cross-Attention
                   |
                   v
        Token + Position Information
                   |
                   v
         Causal Self-Attention
                   |
                   v
          Cross-Attention ← Encoder Output
                   |
                   v
          Feed-Forward Network
                   |
                   v
             Output Projection
                   |
                   v
                  Token

And the three attention patterns are:

text
ENCODER SELF-ATTENTION

Q ← Encoder
K ← Encoder
V ← Encoder

Non-causal
→ source positions can generally attend to one another.


DECODER SELF-ATTENTION

Q ← Decoder
K ← Decoder
V ← Decoder

Causal
→ cannot attend to future target positions.


CROSS-ATTENTION

Q ← Decoder
K ← Encoder
V ← Encoder

→ decoder retrieves relevant information from encoder output.

The core Transformer block can therefore be remembered as:

text
Attention
    ↓
Residual Connection
    ↓
Layer Normalization
    ↓
Feed-Forward Network
    ↓
Residual Connection
    ↓
Layer Normalization

with the decoder additionally containing:

text
Causal Self-Attention
        ↓
Cross-Attention
        ↓
Feed-Forward Network

Knowledge Checkpoint

18. Transformers Architecture Checkpoint

Q1.What is the formula for Scaled Dot-Product Attention in the Transformer architecture?

AAttention(Q, K, V) = Softmax((Q · K^T) / sqrt(d_k)) · V

BAttention(Q, K, V) = (Q · K · V) / d_k

CAttention(Q, K, V) = tanh(Q · K^T) ⊙ V

DAttention(Q, K, V) = Softmax(Q + K) · V

Q2.Why is Positional Encoding necessary in Transformer models?

ASelf-attention is permutation-invariant: without positional encodings, the model treats 'dog bites man' and 'man bites dog' identically.

BTo scale token vectors between 0 and 1.

CTo compress vocabulary sizes.

DTo speed up GPU matrix multiplication.

Q3.Why does Multi-Head Attention project Q, K, V into multiple subspaces (e.g. h=8 or 16 heads)?

AIt allows the model to simultaneously attend to information from different representation subspaces and positions (e.g. syntactic vs semantic relations).

BIt reduces overall computation by 8x.

CIt eliminates the feedforward network layer.

DIt replaces tokenization.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

17. Attention Mechanisms (Bahdanau, Luong, Scaled Dot-Product)

Next Lesson

19. Transformer Families (BERT, GPT, T5, LLaMA & Decoder-Only)