Advanced

24 min read

#Attention Mechanism#Bahdanau#Luong#Query Key Value#Alignment#Seq2Seq

17. Attention Mechanisms (Bahdanau, Luong, Scaled Dot-Product)

The breakthrough in neural representations: Additive (Bahdanau) attention, multiplicative (Luong) attention, and Query-Key-Value mathematical formulation.

Attention Mechanism: Complete Notes (Beginner to Advanced)

Introduction#

The Attention Mechanism is a neural network technique that allows a model to dynamically focus on the most relevant parts of its input when producing a representation or prediction.

Traditional sequence models such as RNNs process information step by step:

›x₁ → x₂ → x₃ → x₄

When processing a particular element, an RNN primarily relies on its current hidden state to carry information from earlier elements.

Attention provides a different mechanism:

text
Current representation
        ↓
Compare with available representations
        ↓
Calculate relevance scores
        ↓
Convert scores into weights
        ↓
Combine information
        ↓
Context-aware representation

The fundamental idea is:

text
Not every piece of information is equally important.
Attention learns which information to focus on.

The main concepts are:

text
Query
Key
Value
Attention Scores
Scaled Dot-Product Attention
Self-Attention
Cross-Attention
Multi-Head Attention
Attention Mask
Causal Mask

1. Attention

Attention allows a model to assign different importance to different input elements.

Suppose a sequence contains:

›x₁  x₂  x₃  x₄

and the model is currently interested in information related to:

›x₃

Instead of treating every element equally, attention can calculate weights such as:

text
x₁ → 0.05
x₂ → 0.10
x₃ → 0.75
x₄ → 0.10

The model then uses these weights to create a weighted combination of information.

Conceptually:

Architecture & Data Flow
Input representations
       |
       v
Relevance calculation
       |
       v
Attention weights
       |
       v
Weighted information
       |
       v
Attention output

Why Attention Is Useful#

Attention helps a model:

Focus on relevant information
Model relationships between different positions
Avoid relying on only a single fixed-size representation of an entire sequence
Capture long-range relationships more directly

For example, in:

›"The animal didn't cross the road because it was tired."

understanding what "it" refers to requires considering other words in the sequence.

Attention can assign higher relevance to useful contextual words.

2. Query

A Query represents the information that is currently looking for relevant information.

It can be thought of as the question:

›"What information am I looking for?"

In attention, the query is represented by:

›Q

Suppose a sequence contains:

›The cat sat on the mat

When processing one position, its query is used to compare that position against available keys.

Conceptually:

Architecture & Data Flow
Query
  |
  v
"What information is relevant to me?"
  |
  v
Compare against Keys

Query in Mathematical Form#

Queries are usually created using a learned projection:

Mathematical Formulation
Q = XW_Q

Where:

Mathematical Formulation
X   = input representations
W_Q = learned query projection matrix
Q   = query representations

For a particular query:

›qᵢ

the model compares it with keys from the available representations.

3. Key

A Key represents information that can be used to determine whether a particular input is relevant to a query.

It can be thought of as an identifier or description:

›"What kind of information do I contain?"

Keys are represented by:

›K

and are commonly calculated as:

Mathematical Formulation
K = XW_K

Where:

Mathematical Formulation
X   = input representations
W_K = learned key projection matrix
K   = key representations

Query-Key Relationship#

Attention compares:

›Query ↔ Key

to determine relevance.

Conceptually:

Architecture & Data Flow
Query
  |
  +------> Key 1 → relevance
  |
  +------> Key 2 → relevance
  |
  +------> Key 3 → relevance
  |
  +------> Key 4 → relevance

A higher similarity generally means:

›This key is more relevant to this query.

4. Value

A Value is the actual information that attention retrieves and combines after determining relevance.

Values are represented by:

›V

and commonly calculated as:

Mathematical Formulation
V = XW_V

Where:

Mathematical Formulation
X   = input representations
W_V = learned value projection matrix
V   = value representations

Query, Key, Value#

The easiest mental model is:

text
Query
→ What am I looking for?

Key
→ What information do I contain?

Value
→ What information should I return if I am relevant?

For example, imagine searching a database:

text
Query → search request
Key   → searchable description
Value → actual stored information

Attention works conceptually in a similar way.

Important Distinction#

The model uses:

›Q and K

to determine:

›how relevant something is

and then uses:

›V

to determine:

›what information is actually aggregated.

5. Attention Scores

Attention Scores measure the compatibility between a query and keys.

For dot-product attention, a query q and key k can be compared using:

Mathematical Formulation
score(q, k) = q · k

For multiple queries and keys:

Mathematical Formulation
Scores = QKᵀ

The result is a matrix in which each entry represents the compatibility between a query and a key.

Example#

Suppose:

Mathematical Formulation
Q =
[q₁]
[q₂]

and:

Mathematical Formulation
K =
[k₁]
[k₂]
[k₃]

Then:

›QKᵀ

produces:

text
          k₁    k₂    k₃

q₁       s₁₁   s₁₂   s₁₃
q₂       s₂₁   s₂₂   s₂₃

Each row corresponds to one query.

From Scores to Weights#

Raw scores are not yet normalized attention weights.

A softmax is commonly applied:

Mathematical Formulation
Attention Weights = softmax(Scores)

For one query:

›Scores:
[2.0, 1.0, 0.0]

softmax converts them into positive values that sum to 1.

Conceptually:

›[0.67, 0.24, 0.09]

The exact values depend on the scores.

Weighted Values#

The resulting weights are used to combine values:

Mathematical Formulation
Attention Output
=
Σ attention_weight × value

Therefore:

text
Scores
   ↓
Softmax
   ↓
Attention Weights
   ↓
Weighted Sum of Values
   ↓
Output

6. Scaled Dot-Product Attention

Scaled Dot-Product Attention is the standard attention operation used in Transformer architectures.

Its equation is:

Mathematical Formulation
Attention(Q, K, V)
=
softmax(QKᵀ / √dₖ)V

Where:

Mathematical Formulation
Q  = Query matrix
K  = Key matrix
V  = Value matrix
dₖ = dimension of the key vectors

Step 1: Calculate Query-Key Scores#

›QKᵀ

This measures compatibility between queries and keys.

Step 2: Scale the Scores#

The scores are divided by:

›√dₖ

So:

›QKᵀ / √dₖ

Why Scale?#

As the dimensionality of key/query vectors increases, dot products can become large in magnitude.

Large logits can cause softmax to become very peaked, which can lead to less useful gradients during training.

Scaling by:

›1 / √dₖ

helps keep the magnitude of the scores more controlled.

Step 3: Apply Softmax#

›softmax(QKᵀ / √dₖ)

This converts scores into normalized attention weights.

Step 4: Multiply by Values#

›softmax(QKᵀ / √dₖ)V

The weights determine how much information is taken from each value.

Complete Flow#

Architecture & Data Flow
             Q
             |
             |
             v
          QKᵀ
             ^
             |
             K
             |
             v
       Scale by √dₖ
             |
             v
          Softmax
             |
             v
    Attention Weights
             |
             ×
             |
             V
             |
             v
      Attention Output

7. Numerical Scaled Dot-Product Example

Suppose there is one query and three keys.

After calculating the dot products:

Mathematical Formulation
QKᵀ = [2, 1, 0]

Assume:

Mathematical Formulation
dₖ = 4

Then:

Mathematical Formulation
√dₖ = 2

Scale the scores:

Mathematical Formulation
[2, 1, 0] / 2
=
[1, 0.5, 0]

Apply softmax:

›softmax([1, 0.5, 0])

Approximately:

›[0.506, 0.307, 0.186]

These are the attention weights.

Suppose the values are:

Mathematical Formulation
v₁ = [1, 0]
v₂ = [0, 1]
v₃ = [1, 1]

Then the output is:

text
0.506[1, 0]
+
0.307[0, 1]
+
0.186[1, 1]

Therefore:

Mathematical Formulation
Output
=
[0.506, 0]
+
[0, 0.307]
+
[0.186, 0.186]

Mathematical Formulation
Output
≈
[0.692, 0.493]

The output is a weighted combination of the values.

8. Self-Attention

Self-Attention is attention in which queries, keys, and values are derived from the same input sequence.

Given:

›X

the model creates:

Mathematical Formulation
Q = XW_Q
K = XW_K
V = XW_V

Then:

Mathematical Formulation
Attention(X)
=
softmax(QKᵀ / √dₖ)V

Why "Self"?#

Because the sequence attends to itself.

For example:

text
Input:

"The cat sat on the mat"

        ↓

Every token can attend to
other tokens in the same sequence.

Conceptually:

text
Token 1 → Token 1, Token 2, Token 3, ...
Token 2 → Token 1, Token 2, Token 3, ...
Token 3 → Token 1, Token 2, Token 3, ...

Self-Attention Matrix#

For a sequence of four tokens:

text
       Key positions
       1    2    3    4

Q1    .    .    .    .
Q2    .    .    .    .
Q3    .    .    .    .
Q4    .    .    .    .

Each cell represents how strongly one query attends to one key.

After softmax, each row contains attention weights.

Why Self-Attention Is Powerful#

Self-attention allows every position to directly interact with every other position within the permitted attention range.

For example:

text
Long sequence

x₁ --------------------→ x₁₀₀

A relationship between distant positions can be modeled through a direct attention interaction rather than requiring information to travel through many recurrent steps.

9. Cross-Attention

Cross-Attention occurs when queries come from one sequence or representation while keys and values come from another.

For example:

text
Q ← Decoder
K ← Encoder
V ← Encoder

Then:

Mathematical Formulation
Attention(Q, K, V)
=
softmax(QKᵀ / √dₖ)V

Self-Attention vs Cross-Attention#

Self-attention:

text
Q, K, V
   ↓
same source

Cross-attention:

text
Q
↓
one source

K, V
↓
another source

Example: Encoder-Decoder Model#

Suppose an encoder processes an input:

text
Source sequence
      ↓
    Encoder
      ↓
Encoder representations

A decoder generates an output sequence.

The decoder can use cross-attention:

Architecture & Data Flow
Decoder representation
        ↓
        Q
        |
        +------+
               |
Encoder → K, V |
               ↓
        Cross-Attention
               ↓
      Context-aware Decoder

This allows the decoder to retrieve relevant information from the encoder's representations.

Example#

In machine translation:

text
English:
"I like cats"

        ↓
     Encoder

        ↓
Representations

        ↓
     Decoder

        ↓
French:
"J'aime les chats"

During decoding, cross-attention helps the decoder focus on relevant encoder representations.

10. Multi-Head Attention

Multi-Head Attention runs multiple attention operations, called heads, in parallel.

Instead of using one attention transformation:

text
Input
 ↓
One Attention
 ↓
Output

the model uses several:

Architecture & Data Flow
             Input
               |
      +--------+--------+
      |        |        |
      v        v        v
    Head 1   Head 2   Head 3
      |        |        |
      +--------+--------+
               |
               v
          Concatenate
               |
               v
        Output Projection
               |
               v
             Output

Why Multiple Heads?#

Different attention heads can learn different relationships.

For example, in a language task, different heads may learn to focus on patterns involving:

text
Syntactic relationships
Word dependencies
Local context
Long-range relationships
Other learned patterns

These are illustrative possibilities, not guaranteed roles assigned to specific heads.

Mathematical Form#

For each head:

Mathematical Formulation
headᵢ =
Attention(QW_Q⁽ⁱ⁾, KW_K⁽ⁱ⁾, VW_V⁽ⁱ⁾)

The heads are concatenated:

Mathematical Formulation
MultiHead(Q,K,V)
=
Concat(head₁, head₂, ..., head_h)W_O

Where:

Mathematical Formulation
W_O = output projection matrix
h   = number of attention heads

Complete Flow#

Architecture & Data Flow
Q, K, V
  |
  +--------+--------+--------+
  |        |        |        |
  v        v        v        v
 Head 1   Head 2   Head 3   ... Head h
  |        |        |             |
  +--------+--------+-------------+
           |
           v
       Concatenate
           |
           v
    Output Projection
           |
           v
         Output

11. Attention Mask

An Attention Mask controls which positions are allowed to participate in attention.

It can be used to prevent the model from attending to certain positions.

Conceptually:

text
Allowed position
→ attention can be calculated

Masked position
→ attention is prevented

A mask is often applied to the attention scores before softmax.

Conceptually:

text
Raw Scores
    ↓
Apply Mask
    ↓
Masked Scores
    ↓
Softmax
    ↓
Attention Weights

Why Mask?#

Masks can be used for several reasons.

Examples include:

text
Padding Mask
→ prevent attention to padding tokens

Causal Mask
→ prevent access to future tokens

The exact mask structure depends on the task.

12. Causal Mask

A Causal Mask is a type of attention mask that prevents a position from attending to future positions.

It is especially important in autoregressive sequence generation.

Suppose the sequence is:

›x₁ x₂ x₃ x₄

At position 2, the model should be allowed to use:

›x₁
x₂

but not:

›x₃
x₄

Causal Attention Pattern#

For four positions:

text
        Keys
        1  2  3  4

Q1      ✓  ✗  ✗  ✗
Q2      ✓  ✓  ✗  ✗
Q3      ✓  ✓  ✓  ✗
Q4      ✓  ✓  ✓  ✓

This creates a lower-triangular attention pattern.

Score Masking#

Before softmax, disallowed positions are typically assigned a very large negative value, conceptually:

text
Allowed:
normal score

Masked:
-∞

For example:

text
Raw scores:

[2.0, 1.0, 0.5, 0.2]

For the first position, a causal mask can produce:

›[2.0, -∞, -∞, -∞]

After softmax:

›[1.0, 0.0, 0.0, 0.0]

Therefore, the first position cannot attend to future positions.

Why Causal Mask Is Important#

Without causal masking, an autoregressive model could use future information while predicting the current token.

That would cause information leakage.

The causal constraint is:

text
Prediction at time t
→ may use positions ≤ t
→ cannot use positions > t

For next-token prediction, implementations may shift inputs and targets so the model predicts the next token while maintaining this causal restriction.

13. Attention Mask vs Causal Mask

A causal mask is one specific type of attention mask.

Architecture & Data Flow
Attention Mask
       |
       +--> Padding Mask
       |
       +--> Causal Mask
       |
       +--> Other task-specific masks

Comparison#

Feature	Attention Mask	Causal Mask
General concept	Controls allowed attention connections	Prevents attention to future positions
Main purpose	Depends on task	Preserve temporal/causal ordering
Used for padding	Yes	No, not its primary purpose
Blocks future tokens	Sometimes	Yes
Common in autoregressive generation	Can be	Yes

14. Query-Key-Value Flow

The entire attention mechanism can be understood as:

Architecture & Data Flow
Input Representations
        |
        +------------------+
        |                  |
        v                  v
      Query              Key
        |                  |
        +--------+---------+
                 |
                 v
          Attention Scores
                 |
                 v
             Scaling
                 |
                 v
              Masking
                 |
                 v
              Softmax
                 |
                 v
        Attention Weights
                 |
                 |
                 v
              Value
                 |
                 v
       Weighted Combination
                 |
                 v
          Attention Output

The three fundamental roles are:

text
Q → What am I looking for?

K → Where is relevant information?

V → What information should I retrieve?

15. Self-Attention vs Cross-Attention

Feature	Self-Attention	Cross-Attention
Query source	Same sequence	One representation/source
Key source	Same sequence	Another representation/source
Value source	Same sequence	Another representation/source
Main purpose	Model relationships within a sequence	Retrieve information from another representation
Common example	Transformer encoder/self-attention	Encoder-decoder attention

The mathematical operation is the same:

›softmax(QKᵀ / √dₖ)V

The important difference is where:

text
Q
K
V

come from.

16. Single-Head vs Multi-Head Attention

Single-Head Attention#

text
Q, K, V
   ↓
Attention
   ↓
Output

Multi-Head Attention#

Architecture & Data Flow
Q, K, V
   |
   +--> Head 1
   +--> Head 2
   +--> Head 3
   +--> ...
   +--> Head h
             |
             v
        Concatenate
             |
             v
       Output Projection
             |
             v
           Output

The advantage of multiple heads is that the model can learn multiple attention patterns in parallel.

17. Simple Scaled Dot-Product Attention with PyTorch

A basic implementation can be written as:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn.functional as F

Q = torch.randn(2, 4, 8)
K = torch.randn(2, 4, 8)
V = torch.randn(2, 4, 8)

d_k = Q.size(-1)

scores = Q @ K.transpose(-2, -1)

scores = scores / (d_k ** 0.5)

weights = F.softmax(scores, dim=-1)

output = weights @ V

print("Scores:", scores.shape)
print("Weights:", weights.shape)
print("Output:", output.shape)

For:

Mathematical Formulation
batch = 2
sequence length = 4
dₖ = 8

the shapes are:

Mathematical Formulation
Q = (2, 4, 8)
K = (2, 4, 8)
V = (2, 4, 8)

QKᵀ = (2, 4, 4)

Output = (2, 4, 8)

The 4 × 4 matrix represents interactions between every query position and every key position.

18. Causal Mask in PyTorch

A causal mask can be created using a triangular matrix.


🐍 PythonInteractive WebAssembly
import torch

sequence_length = 4

mask = torch.tril(
    torch.ones(sequence_length, sequence_length)
)

print(mask)

Conceptually:

text
[[1, 0, 0, 0],
 [1, 1, 0, 0],
 [1, 1, 1, 0],
 [1, 1, 1, 1]]

The zeros indicate positions that should not be attended to.

One common implementation converts those positions to a large negative score before softmax:


🐍 PythonInteractive WebAssembly
scores = scores.masked_fill(mask == 0, float("-inf"))

weights = torch.softmax(scores, dim=-1)

The result is that future positions receive zero attention probability after softmax.

19. Important Terminology

Term	Meaning
Attention	Mechanism for dynamically weighting information
Query	Representation asking what information is relevant
Key	Representation used to determine relevance
Value	Information that is aggregated
Attention Score	Compatibility between a query and a key
Scaled Dot-Product Attention	`softmax(QKᵀ / √dₖ)V`
Self-Attention	Q, K, V derived from the same source
Cross-Attention	Q comes from one source; K and V come from another
Multi-Head Attention	Multiple attention operations performed in parallel
Attention Mask	Restricts which positions can be attended to
Causal Mask	Prevents attention to future positions

20. Summary

Concept	Core Idea
Attention	Focus on relevant information
Query	What information am I looking for?
Key	How can my information be matched to a query?
Value	What information should be retrieved?
Attention Scores	Measure query-key compatibility
Scaled Dot-Product	Computes normalized weighted information using Q, K, V
Self-Attention	Sequence attends to itself
Cross-Attention	One representation attends to another
Multi-Head Attention	Multiple attention patterns learned in parallel
Attention Mask	Restricts attention connections
Causal Mask	Blocks future information

21. Quick Recap

text
Attention
→ Dynamically focuses on relevant information.

Query
→ What am I looking for?

Key
→ What information do I contain for matching?

Value
→ What information should be retrieved?

Attention Score
→ How relevant is a key to a query?

Scaled Dot-Product Attention
→ softmax(QKᵀ / √dₖ)V

Self-Attention
→ Q, K, V come from the same source.

Cross-Attention
→ Q comes from one source and K, V from another.

Multi-Head Attention
→ Multiple attention operations run in parallel.

Attention Mask
→ Controls which positions may participate in attention.

Causal Mask
→ Prevents a position from seeing future positions.

Final Mental Model

Architecture & Data Flow
                    ATTENTION
                        |
                        v
                  +-----------+
                  |     Q     |
                  |     K     |
                  |     V     |
                  +-----------+
                        |
                        v
                   QKᵀ Scores
                        |
                        v
                  Scale by √dₖ
                        |
                        v
                  Apply Mask
                        |
                        v
                     Softmax
                        |
                        v
                Attention Weights
                        |
                        ×
                        |
                        v
                       V
                        |
                        v
               Weighted Combination
                        |
                        v
                     Output

And the most important equation is:

Mathematical Formulation
Attention(Q, K, V)
=
softmax(QKᵀ / √dₖ)V

The key distinction to remember is:

text
Q + K
→ determine WHERE to focus

V
→ determine WHAT information is retrieved

For self-attention:

›Same source → Q, K, V

For cross-attention:

›Source A → Q
Source B → K, V

For causal attention:

text
Current position
      ↓
Can attend to:
past + current

Cannot attend to:
future

Knowledge Checkpoint

17. Attention Mechanism Checkpoint

Q1.What fundamental bottleneck in Seq2Seq encoder-decoder RNNs motivated the creation of the Attention Mechanism?

AThe fixed-size context vector bottleneck: forcing an entire input sentence of arbitrary length into a single fixed-dimensional vector h_T.

BRNNs running out of GPU memory on batch size 1.

CThe inability of Softmax to compute probabilities.

DThe requirement for labeled datasets.

Q2.In Query-Key-Value attention formulation, how is the context vector computed?

AAttention weights alpha = Softmax(Score(Q, K)), and Context = sum(alpha_i * V_i).

BContext = Q · K · V

CContext = Q + K + V

DContext = Softmax(Q) · (K - V)

Q3.How do Bahdanau (Additive) and Luong (Multiplicative) attention score functions differ?

ABahdanau uses a feedforward network Score(s, h) = v^T · tanh(W_s s + W_h h), whereas Luong uses matrix dot-products Score(s, h) = s^T · W · h.

BBahdanau only works for speech, Luong only for images.

CLuong attention does not use Softmax.

DBahdanau attention does not compute gradients.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

16. Gated Recurrent Units (GRU)

Next Lesson

18. Transformers Architecture & Multi-Head Self-Attention