Intermediate

18 min read

#Embeddings#Word2Vec#GloVe#Vector Space#Cosine Similarity#Tokenization

20. High-Dimensional Embeddings & Vector Representations

Mapping discrete entities to dense continuous vector spaces: One-hot vs dense vectors, Word2Vec (Skip-gram & CBOW), subword tokenization, and vector similarity metrics.

Embeddings: Complete Notes (Beginner to Advanced)

1. Embeddings#

An embedding is a numerical vector representation of data such as words, tokens, sentences, or other objects.

Instead of representing language as discrete symbols:

text
"cat"
"dog"
"king"
"queen"

an embedding represents each item as a vector:

Architecture & Data Flow
cat   -> [0.21, -0.44, 0.73, ...]
dog   -> [0.18, -0.39, 0.69, ...]
king  -> [-0.12, 0.81, 0.24, ...]

The goal is to represent information in a form that neural networks can process and learn from.

Basic Idea#

Architecture & Data Flow
Text / Object
     |
     v
Embedding Model
     |
     v
Numerical Vector
     |
     v
Neural Network / Similarity / Retrieval

An embedding is therefore a learned representation of an object in a continuous vector space.

2. Why Do We Need Embeddings?#

Neural networks operate on numerical values.

A word such as:

›"cat"

cannot be directly multiplied by a weight matrix.

A simple integer encoding would be:

Architecture & Data Flow
cat -> 1
dog -> 2
car -> 3

but this creates a false interpretation:

text
dog (2) is closer to cat (1)
than
car (3)

The numbers themselves do not represent semantic relationships.

Embeddings instead represent objects using vectors whose dimensions can capture useful learned patterns.

Architecture & Data Flow
cat -> [0.2, 0.7, -0.1, ...]
dog -> [0.3, 0.6, -0.2, ...]
car -> [-0.8, 0.1, 0.5, ...]

Similar objects can be represented by vectors that are close according to a chosen similarity or distance measure.

3. Word Embeddings#

A word embedding represents a word as a dense numerical vector.

Examples of classic word-embedding methods include:

Word2Vec
GloVe
FastText

Conceptually:

Architecture & Data Flow
king  -> vector
queen -> vector
man   -> vector
woman -> vector

The vectors are learned from patterns in text.

Distributional Idea#

Word embeddings are based on the idea that:

Words appearing in similar contexts tend to have similar meanings.

For example:

›The cat drank milk.
The dog drank milk.

The words cat and dog occur in similar contexts, so a learned embedding can place them relatively close in vector space.

4. Word2Vec#

Word2Vec is a family of methods for learning word embeddings.

Two classic training approaches are:

Architecture & Data Flow
Word2Vec
   |
   +-- CBOW
   |
   +-- Skip-gram

4.1 CBOW#

Continuous Bag of Words (CBOW) predicts a target word from surrounding context words.

Example:

›The cat drank milk

If the target is:

›drank

the context may be:

›cat, milk

Conceptually:

Architecture & Data Flow
Context words
     |
     v
   CBOW
     |
     v
Target word

4.2 Skip-gram#

Skip-gram does the opposite.

It uses a target word to predict surrounding context words.

Architecture & Data Flow
Target word
     |
     v
 Skip-gram
     |
     v
Context words

Example:

text
Target: cat

Predict:
the
sat
on
...

5. Token Embeddings#

A token embedding represents a token rather than necessarily a complete word.

This distinction is important because modern NLP models usually tokenize text into tokens, and a token may be:

a complete word
part of a word
punctuation
a special token

For example, a tokenizer might represent:

›unhappiness

as:

›["un", "happiness"]

or use another subword representation depending on the tokenizer.

Each token receives a vector:

›"un"        -> [ ... ]
"happiness" -> [ ... ]

Token Embedding Layer#

In a Transformer, token IDs are passed to an embedding table.

Architecture & Data Flow
Token IDs
   |
   v
Embedding Matrix
   |
   v
Token Vectors

Suppose the vocabulary size is:

Mathematical Formulation
V = 50,000

and embedding dimension is:

Mathematical Formulation
D = 768

Then the embedding matrix has shape:

›(50,000, 768)

For token ID 125, the model retrieves row 125.

Architecture & Data Flow
Token ID 125
     |
     v
Embedding Matrix
     |
     v
Vector of length 768

Important Point#

The embedding layer is usually a learnable parameter matrix.

6. Word Embeddings vs Token Embeddings#

Feature	Word Embedding	Token Embedding
Unit	Word	Token
Tokenization required?	Not necessarily	Yes
Can represent subwords?	Usually no	Yes
Common in modern Transformers?	Less directly	Yes
Example	`playing`	`play` + `##ing`

A token embedding is therefore more general than a traditional word embedding.

7. Sentence Embeddings#

A sentence embedding represents an entire sentence as a single vector.

Example:

›"The cat is sleeping."

might become:

›[0.12, -0.44, 0.73, ..., 0.08]

The vector attempts to capture useful information about the sentence's meaning.

Sentence Embedding Pipeline#

Architecture & Data Flow
Sentence
   |
Tokenization
   |
Encoder
   |
Token representations
   |
Pooling / representation selection
   |
Sentence vector

Applications#

Sentence embeddings are useful for:

Semantic search
Similarity comparison
Clustering
Duplicate detection
Retrieval
Recommendation
Classification

Example#

Consider:

text
A: "The dog is sleeping."
B: "A dog is taking a nap."
C: "The stock market increased today."

A good semantic embedding model should generally place:

›A and B -> closer
A and C -> farther apart

according to an appropriate similarity measure.

8. Contextual Embeddings#

A contextual embedding is a representation whose value depends on the surrounding context.

Traditional word embeddings generally assign one vector to a word regardless of where it appears.

For example:

›bank -> one fixed vector

But bank has different meanings:

›I deposited money in the bank.

and:

›We sat beside the river bank.

A contextual model can produce different representations for bank in these two contexts.

Architecture & Data Flow
Sentence 1:
money -> bank -> deposit
             |
             v
       Contextual vector A

Sentence 2:
river -> bank -> beside
             |
             v
       Contextual vector B

Contextual Embeddings in Transformers#

Transformer models use self-attention to incorporate information from surrounding tokens.

Architecture & Data Flow
Input tokens
     |
     v
Token embeddings
     |
     v
Transformer layers
     |
     v
Contextual token representations

Therefore, the representation of a token after passing through Transformer layers is contextualized.

Static vs Contextual#

Feature	Static Embedding	Contextual Embedding
Vector for same word	Usually same	Can change by context
Context considered	Limited during representation lookup	Yes
Example	Word2Vec	BERT representations
Handles polysemy	Poorly	Better

9. Positional Embeddings#

Transformers use attention, but attention alone does not inherently provide information about the order of tokens.

Consider:

›Dog bites man.

and:

›Man bites dog.

The same tokens are present, but their order changes the meaning.

The model therefore needs positional information.

A common approach is:

Architecture & Data Flow
Token Embedding + Positional Representation
                    |
                    v
              Transformer

Conceptual Example#

Architecture & Data Flow
Token:       "cat"
Token vector:       [ ... ]

Position:       3
Position vector:   [ ... ]

                    +
                    |
                    v

Input representation [ ... ]

10. Positional Encoding vs Positional Embedding#

These terms are related but should not always be treated as identical.

Positional Encoding#

A positional encoding is a predefined function that generates a representation for each position.

The original Transformer used sinusoidal positional encodings:

Mathematical Formulation
PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))

PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

These values are combined with token representations.

Positional Embedding#

A positional embedding is a learned vector associated with a position.

Conceptually:

Architecture & Data Flow
Position 0 -> learned vector
Position 1 -> learned vector
Position 2 -> learned vector
...

The model learns these vectors during training.

Important Point#

Modern Transformer architectures can use several different approaches to represent position, including learned positional embeddings and other positional mechanisms.

11. Embedding Space#

The embedding space is the mathematical vector space in which embeddings are represented.

Suppose each object is represented by a vector with three dimensions:

Architecture & Data Flow
cat  -> [0.8, 0.2, 0.1]
dog  -> [0.7, 0.3, 0.2]
car  -> [-0.5, 0.8, 0.9]

These vectors occupy different locations in a 3D space.

For real models, embedding dimensions can be hundreds or thousands, so we cannot directly visualize the full space.

Conceptual Visualization#

The important idea is not the literal visual distance in a 2D drawing, but the mathematical relationship between vectors.

12. Similarity in Embedding Space#

To determine whether two embeddings are similar, we can use a similarity or distance measure.

Cosine Similarity#

A common measure for embeddings is cosine similarity:

Mathematical Formulation
cosine_similarity(A, B)
=
(A · B) / (||A|| ||B||)

where:

Mathematical Formulation
A · B = dot product
||A|| = magnitude of A
||B|| = magnitude of B

The value is typically in the range:

›-1 to 1

for unrestricted real-valued vectors.

Higher cosine similarity generally indicates that the vectors point in more similar directions.

Example#

Architecture & Data Flow
Embedding A -> "The dog is sleeping."
Embedding B -> "A dog is taking a nap."

Cosine similarity -> high

while:

Architecture & Data Flow
Embedding A -> "The dog is sleeping."
Embedding C -> "The stock market rose."

Cosine similarity -> lower

The exact value depends on the embedding model.

13. Embedding Space and Semantic Relationships#

A well-trained embedding space can organize related concepts in useful ways.

For example:

text
cat
dog
horse

may form a region associated with animals.

Other concepts may form different regions:

text
car
bus
train

Important Caution#

Embedding dimensions are not necessarily human-interpretable.

It is usually incorrect to assume:

Mathematical Formulation
dimension 1 = intelligence
dimension 2 = gender
dimension 3 = size

Instead, semantic information is often distributed across many dimensions.

The useful structure emerges from relationships between complete vectors.

14. Representation Learning#

Representation learning is the process of allowing a machine-learning model to learn useful representations of raw data automatically.

Instead of manually defining features:

text
Raw data
   |
Manually engineered features
   |
Model

representation learning aims for:

text
Raw data
   |
Neural network
   |
Learned representation
   |
Task

Example: Image#

Traditional approach:

text
Image
 |
Manually calculate edges, shapes, textures
 |
Classifier

Deep learning approach:

text
Image
 |
Neural network
 |
Learned features
 |
Classifier

Example: Language#

Instead of manually assigning semantic features to words:

Architecture & Data Flow
cat -> animal = 1
cat -> vehicle = 0
...

the model can learn vector representations from data.

15. Embeddings as Learned Representations#

Embeddings are one important form of representation learning.

For example:

Architecture & Data Flow
Text
 |
Tokenization
 |
Token IDs
 |
Embedding Layer
 |
Dense vectors
 |
Transformer
 |
Contextual representations

During training, the model updates its parameters so that the representations become useful for the objective.

Important Distinction#

A token embedding from an embedding lookup table and a contextual representation produced after multiple Transformer layers are not necessarily the same thing.

Architecture & Data Flow
Token ID
   |
   v
Token Embedding
   |
   v
Transformer layers
   |
   v
Contextual Representation

The first is a learned lookup representation.

The second incorporates information from the surrounding sequence.

16. Embedding Dimensions#

An embedding has a fixed number of dimensions.

For example:

Architecture & Data Flow
Embedding dimension = 4

"cat" ->
[0.2, 0.8, -0.1, 0.4]

A larger dimension provides more capacity to encode information, but it also increases:

Memory usage
Computational cost
Number of parameters in an embedding table

Embedding Matrix#

If:

Mathematical Formulation
Vocabulary size = V
Embedding dimension = D

then:

Mathematical Formulation
Embedding matrix shape = (V, D)

If the embedding matrix is trainable and has no additional parameters beyond the vectors:

Mathematical Formulation
Number of parameters = V × D

Example:

Mathematical Formulation
V = 10,000
D = 300

Parameters = 10,000 × 300
           = 3,000,000

17. Embedding Lookup#

Suppose the vocabulary contains:

Architecture & Data Flow
0 -> the
1 -> cat
2 -> dog
3 -> runs

and the embedding matrix is:

Mathematical Formulation
E =
[
  vector_0
  vector_1
  vector_2
  vector_3
]

For token ID 2:

›E[2]

returns the embedding for:

›dog

Conceptually:

Architecture & Data Flow
Token ID 2
    |
    v
Embedding Matrix
    |
    v
Dog embedding

The model does not need to calculate a new vector from scratch for every occurrence of the token during the lookup.

18. Simple Embedding Example with PyTorch#

A trainable embedding layer can be created using nn.Embedding.


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

vocab_size = 10_000
embedding_dim = 128

embedding = nn.Embedding(vocab_size, embedding_dim)

token_ids = torch.tensor([1, 25, 900])

vectors = embedding(token_ids)

print(vectors.shape)

Output:

›torch.Size([3, 128])

Each token ID is mapped to a vector of length 128.

19. Sentence Embedding Concept#

A Transformer produces one vector per token:

Architecture & Data Flow
Token 1 -> vector
Token 2 -> vector
Token 3 -> vector
...
Token T -> vector

To obtain one vector for the entire sentence, a model can use an appropriate sentence-level representation or a pooling operation.

For example, mean pooling:

text
Sentence:
x1, x2, x3, ..., xT

Sentence embedding:
(x1 + x2 + ... + xT) / T

In practice, high-quality sentence embeddings are often produced by models trained specifically for sentence-level semantic tasks rather than simply averaging arbitrary token embeddings.

20. Word, Token, Sentence, Contextual, and Positional Embeddings#

Type	Represents	Context-dependent?	Main purpose
Word embedding	Word	Usually no	Represent words
Token embedding	Token	Usually no at lookup stage	Convert token IDs to vectors
Sentence embedding	Sentence	Depends on model	Represent sentence meaning
Contextual embedding	Token in context	Yes	Context-aware representation
Positional embedding	Position	Position-dependent	Represent token order

Important Relationship#

These concepts can coexist.

For example:

Architecture & Data Flow
Text
 |
Tokenization
 |
Token IDs
 |
Token Embeddings
 +
Positional Information
 |
Transformer
 |
Contextual Token Representations
 |
Pooling / Sentence Representation
 |
Sentence Embedding

21. Embeddings in Modern NLP Systems#

Embeddings appear at multiple stages of modern NLP systems.

Architecture & Data Flow
Raw Text
   |
   v
Tokenizer
   |
   v
Token IDs
   |
   v
Token Embeddings
   +
Positional Information
   |
   v
Transformer
   |
   v
Contextual Representations
   |
   +----------------------+
   |                      |
   v                      v
Task Head             Embedding Pooling
                           |
                           v
                    Sentence Embedding

Embeddings can then be used for:

Classification
Search
Retrieval
Clustering
Recommendation
Similarity
RAG pipelines

22. Summary#

Concept	Simple meaning
Embedding	Numerical vector representation
Word embedding	Vector representing a word
Token embedding	Vector representing a token
Sentence embedding	Vector representing a complete sentence
Contextual embedding	Representation that changes according to context
Positional embedding	Representation of token position
Embedding space	Vector space containing embeddings
Representation learning	Learning useful representations automatically

23. Quick Recap#

Architecture & Data Flow
Word Embedding
    -> Represents a word

Token Embedding
    -> Represents a token ID as a dense vector

Sentence Embedding
    -> Represents an entire sentence as one vector

Contextual Embedding
    -> Represents a token using its surrounding context

Positional Embedding
    -> Represents where a token occurs in a sequence

Embedding Space
    -> Vector space where embeddings live

Representation Learning
    -> Learning useful representations from data

One-Line Mental Model#

Architecture & Data Flow
Raw data
   |
Representation learning
   |
Embeddings
   |
Vector space
   |
Similarity / Retrieval / Neural Network

Knowledge Checkpoint

20. Embeddings Checkpoint

Q1.Why are dense low-dimensional embeddings (e.g. d=768) superior to sparse One-Hot representations (d=50,000+)?

ADense vectors capture continuous semantic similarity in vector space (e.g. Cosine Sim(king - man + woman, queen) is high) and eliminate the curse of dimensionality.

BOne-hot vectors cannot be loaded into CPU RAM.

CDense vectors are always integers.

DOne-hot vectors cause vanishing gradients on GPUs.

Q2.What is the difference between Word2Vec Skip-gram and Continuous Bag of Words (CBOW)?

ACBOW predicts the target center word from surrounding context words, while Skip-Gram predicts surrounding context words given the center word.

BCBOW only works on characters, Skip-Gram on full documents.

CSkip-Gram requires labeled classification targets.

DCBOW is a recurrent neural network.

Q3.Why is Subword Tokenization (BPE, WordPiece, SentencePiece) standard in modern LLM embedding layers?

AIt solves the Out-Of-Vocabulary (OOV) problem by decomposing rare and compound words into frequent subword units while keeping vocabulary size compact (~32k-128k).

BIt converts all text directly into floating point numbers without an embedding lookup table.

CIt eliminates the need for attention matrices.

DIt translates text into bytecode.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

19. Transformer Families (BERT, GPT, T5, LLaMA & Decoder-Only)

Next Lesson

21. Autoencoders & Variational Autoencoders (VAE)