CNN Architectures: Complete Notes (Beginner to Advanced)

---

1. LeNet#

LeNet is one of the earliest successful convolutional neural network architectures. It was designed by Yann LeCun and collaborators for handwritten digit recognition, especially the MNIST-style problem.

The architecture established the fundamental CNN pattern of:

Architecture & Data Flow
Copy
Input
  |
  v
Convolution
  |
  v
Pooling
  |
  v
Convolution
  |
  v
Pooling
  |
  v
Fully Connected Layers
  |
  v
Output

1.1 Core Architecture#

A classic LeNet-5 structure is approximately:

Architecture & Data Flow
Copy
Input
  |
  v
C1: Convolution
  |
  v
S2: Subsampling / Pooling
  |
  v
C3: Convolution
  |
  v
S4: Subsampling / Pooling
  |
  v
C5: Convolution
  |
  v
F6: Fully Connected
  |
  v
Output

The original LeNet-5 used a 32 × 32 input and progressively reduced spatial dimensions while increasing the number of learned feature maps.

1.2 Important Characteristics#

• Early CNN architecture.
• Designed primarily for handwritten character recognition.
• Uses convolution and subsampling layers to learn spatial features.
• Uses fully connected layers near the end.
• Demonstrated that learned convolutional features could be used successfully for visual recognition.

1.3 Importance of LeNet#

LeNet established an important architectural idea:

text
Ask AICopy
Local feature extraction
        +
Spatial downsampling
        +
Classification

This pattern became the foundation for many later CNN architectures.

---

2. AlexNet#

AlexNet was a landmark CNN architecture that achieved a major improvement in large-scale image classification performance in the 2012 ImageNet competition.

It demonstrated the practical power of:

• deeper CNNs
• ReLU activation
• GPU-based training
• dropout
• data augmentation
• max pooling

2.1 Architecture#

A simplified AlexNet flow is:

Architecture & Data Flow
Copy
Input Image
    |
    v
Convolution
    |
    v
ReLU
    |
    v
Max Pooling
    |
    v
Convolution
    |
    v
ReLU
    |
    v
Max Pooling
    |
    v
More Convolution Layers
    |
    v
Fully Connected Layers
    |
    v
Output

The original architecture contained five convolutional layers followed by three fully connected layers.

2.2 ReLU#

AlexNet helped establish ReLU as a practical activation function for deep CNNs.

Instead of:

›sigmoid

the network used:

Mathematical Formulation
Copy
ReLU(x) = max(0, x)

ReLU provided a simple computation and helped make deep network training more effective.

2.3 GPU Training#

AlexNet was trained using GPUs, allowing large convolution operations to be performed much faster than would have been practical on CPUs alone.

This was an important step in the development of modern deep learning.

2.4 Dropout#

AlexNet used dropout in its fully connected layers as a regularization technique.

Conceptually:

text
Ask AICopy
During training:

Neuron  Neuron  Neuron  Neuron
  X       |       X       |

Some activations are randomly dropped during training.

This reduces reliance on particular neurons and can improve generalization.

2.5 Importance of AlexNet#

AlexNet demonstrated that a relatively deep CNN trained with modern computational resources could outperform traditional computer-vision approaches by a large margin on ImageNet.

---

3. VGG#

VGG is a CNN family developed by the Visual Geometry Group at the University of Oxford.

VGG is known for its simple and highly regular architecture.

3.1 Core Design#

The central idea was to use many small:

›3 × 3

convolution kernels.

A simplified structure is:

Architecture & Data Flow
Copy
Input
  |
  v
3×3 Conv
  |
  v
3×3 Conv
  |
  v
Max Pool
  |
  v
3×3 Conv
  |
  v
3×3 Conv
  |
  v
Max Pool
  |
  v
...
  |
  v
Fully Connected Layers
  |
  v
Output

3.2 VGG-16#

One of the best-known variants is VGG-16.

The number 16 refers to the number of learnable layers under the convention used for the architecture:

Mathematical Formulation
Copy
13 convolutional layers
+
3 fully connected layers
=
16 learnable layers

3.3 Why 3 × 3 Convolutions?#

VGG used repeated small convolutions rather than relying heavily on larger kernels.

For example:

›Two 3×3 convolutions

provide an effective receptive field comparable to:

›5×5

while introducing an additional nonlinear transformation between the convolutional layers.

This allows the network to build more complex representations through a sequence of smaller operations.

3.4 Characteristics#

• Very regular architecture.
• Primarily uses 3 × 3 convolutions.
• Uses max pooling for spatial downsampling.
• Becomes progressively deeper across variants.
• Relatively easy to understand structurally.
• Has a large number of parameters, especially in its fully connected layers.

3.5 Importance of VGG#

VGG showed the value of increasing CNN depth while maintaining a simple, repeatable architectural pattern.

It also became widely used as a feature extractor and baseline architecture.

---

4. GoogLeNet#

GoogLeNet, also known as Inception v1, was introduced by Google researchers and won the 2014 ImageNet competition.

Its major architectural contribution was the Inception module.

4.1 Motivation#

Simply making a CNN deeper or wider increases:

›Representational capacity

but can also increase:

text
Ask AICopy
Computation
Memory requirements
Number of parameters

GoogLeNet addressed this by designing modules that perform multiple types of operations in parallel.

4.2 Overall Architecture#

A simplified flow is:

Architecture & Data Flow
Copy
Input
  |
  v
Initial Convolutions
  |
  v
Inception Module
  |
  v
Inception Module
  |
  v
Inception Module
  |
  v
...
  |
  v
Classification

4.3 Auxiliary Classifiers#

The original GoogLeNet included auxiliary classifiers at intermediate depths.

Conceptually:

Architecture & Data Flow
Copy
                -> Auxiliary Classifier
               /
Input -> Layers -> Layers -> Layers -> Main Classifier

These auxiliary branches provided additional training signals during training.

4.4 Importance#

GoogLeNet demonstrated that a network could become deeper while controlling computational cost through a more efficient architecture rather than simply increasing every layer's width.

---

5. Inception#

Inception is the architectural family built around the Inception module.

The central idea is to process the same input using multiple transformations in parallel and then combine their outputs.

5.1 Basic Inception Module#

A simplified Inception module contains parallel branches such as:

Architecture & Data Flow
Copy
                  -> 1×1 Conv --------\
                 /                     \
Input ----------> 3×3 Conv ------------+--> Concatenate
                 \                     /
                  -> 5×5 Conv --------/
                 \
                  -> Pooling ----------/

The branch outputs are concatenated along the channel dimension.

5.2 Why Multiple Branches?#

Different kernel sizes can capture patterns at different spatial scales.

For example:

Architecture & Data Flow
Copy
1 × 1 -> channel mixing / local transformation
3 × 3 -> medium local patterns
5 × 5 -> larger local patterns
Pooling -> summarized local information

The module therefore allows the network to process information at multiple scales.

5.3 1 × 1 Convolution#

In Inception modules, 1 × 1 convolutions are particularly useful for changing the number of channels.

For an input:

›H × W × C

a 1 × 1 convolution can transform it into:

›H × W × C'

while preserving the spatial dimensions when stride and padding are chosen appropriately.

It can therefore be used as a channel projection.

5.4 Factorization and Later Inception Versions#

Later Inception architectures introduced more efficient convolutional designs.

For example, a larger convolution could be factorized into smaller operations:

›5 × 5

can be approximated using:

›3 × 3 -> 3 × 3

Later versions also used asymmetric convolutions such as:

›1 × 3
3 × 1

to reduce computation while retaining a useful receptive field.

5.5 Core Idea#

The key idea behind the Inception family is:

Architecture & Data Flow
Copy
Multiple parallel transformations
              |
              v
Combine their representations
              |
              v
Richer multi-scale representation

---

6. ResNet#

ResNet (Residual Network) introduced residual learning as a practical way to train very deep neural networks.

Its central architectural component is the residual connection.

6.1 Problem Addressed by ResNet#

Simply stacking more layers does not guarantee better performance.

As networks become very deep, optimization can become difficult.

ResNet introduced shortcut paths that allow information to bypass one or more layers.

6.2 Residual Block#

A basic residual block can be represented as:

Architecture & Data Flow
Copy
             ┌───────────────────────┐
             │                       │
x ---------->│ Conv -> ReLU -> Conv  │
|            │                       |
|            └────────────────------>+
|                                    |
└──────────────────────────────────->+
                                     |
                                     v
                                   ReLU
                                     |
                                     v
                                     y

Mathematically:

Mathematical Formulation
Copy
y = F(x) + x

where:

Mathematical Formulation
Copy
F(x) = learned residual transformation
x    = shortcut / identity path

6.3 Projection Shortcut#

If the input and transformed output do not have compatible dimensions, the shortcut can use a projection:

Mathematical Formulation
Copy
y = F(x) + W_s x

where W_s transforms the input to the required shape.

6.4 Why ResNet Works#

The shortcut creates a direct path through the network.

Conceptually:

Architecture & Data Flow
Copy
Without shortcut:

x -> Layer -> Layer -> Layer -> Layer -> y


With shortcut:

x -------------------------------> +
 \-> Layer -> Layer -> Layer ------>

This improves information and gradient flow and makes optimization of very deep networks more practical.

6.5 ResNet Variants#

Common variants include:

text
Ask AICopy
ResNet-18
ResNet-34
ResNet-50
ResNet-101
ResNet-152

The number indicates the approximate depth of the architecture under the counting convention used by the ResNet family.

6.6 Basic vs Bottleneck Blocks#

Shallower ResNets commonly use basic residual blocks.

Deeper variants such as ResNet-50, ResNet-101, and ResNet-152 use bottleneck blocks.

A bottleneck block uses:

Architecture & Data Flow
Copy
1×1 Conv
   |
3×3 Conv
   |
1×1 Conv

The 1 × 1 convolutions reduce and then restore the channel dimension around the central 3 × 3 convolution.

This makes very deep networks more computationally practical.

6.7 ResNet in PyTorch#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torchvision.models as models

model = models.resnet18(weights=None)

print(model)

---

7. DenseNet#

DenseNet (Densely Connected Convolutional Network) connects each layer to many later layers within a dense block.

Instead of only passing information from one layer to the next:

›x1 -> Layer 2 -> Layer 3 -> Layer 4

DenseNet uses connections such as:

Architecture & Data Flow
Copy
x1 ---------------------> Layer 2
 |                         |
 +------------------------> Layer 3
 |                         |
 +------------------------> Layer 4

x2 -----------------------> Layer 3
 |                         |
 +------------------------> Layer 4

x3 -----------------------> Layer 4

7.1 Dense Connectivity#

For a dense block:

Mathematical Formulation
Copy
x0

x1 = H1([x0])

x2 = H2([x0, x1])

x3 = H3([x0, x1, x2])

...

Here:

›[x0, x1, x2]

means concatenation along the channel dimension.

7.2 Concatenation vs Addition#

This is an important distinction.

ResNet:

Mathematical Formulation
Copy
y = F(x) + x

DenseNet:

Mathematical Formulation
Copy
y = [x, F(x)]

where the representations are concatenated rather than added.

Therefore, DenseNet connections preserve previous feature maps directly as part of the new representation.

7.3 Dense Block#

A simplified dense block looks like:

Architecture & Data Flow
Copy
Input
  |
  +------> Layer 1
  |          |
  |          v
  +------> Layer 2
  |          |
  +----------+
  |
  +------> Layer 3

Each layer receives the feature maps from all preceding layers in that dense block.

7.4 Growth Rate#

DenseNet uses a concept called the growth rate.

If:

Mathematical Formulation
Copy
k = growth rate

each layer typically adds approximately k new feature maps.

For example:

Architecture & Data Flow
Copy
Initial channels = 32
Growth rate = 16

After Layer 1 -> 48 channels
After Layer 2 -> 64 channels
After Layer 3 -> 80 channels

because each layer contributes 16 new channels.

7.5 Benefits#

Dense connectivity can:

• encourage feature reuse
• provide strong information flow
• provide direct paths for gradients
• reduce redundant relearning of similar features

The concatenation strategy can, however, increase memory usage because previous feature maps must remain available within the dense block.

7.6 DenseNet in PyTorch#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torchvision.models as models

model = models.densenet121(weights=None)

print(model)

---

8. MobileNet#

MobileNet is a family of CNN architectures designed for efficient computation, especially on mobile and edge devices.

Its key idea is the use of depthwise separable convolution.

8.1 Standard Convolution vs MobileNet Convolution#

A standard convolution simultaneously performs:

text
Ask AICopy
Spatial filtering
+
Channel mixing

MobileNet separates these operations.

Architecture & Data Flow
Copy
Input
  |
  v
Depthwise Convolution
  |
  v
Pointwise Convolution
  |
  v
Output

8.2 Depthwise Convolution#

A depthwise convolution applies one spatial convolution independently to each input channel.

If the input has:

›C channels

then depthwise convolution uses approximately:

›C spatial filters

with each filter operating on one channel.

8.3 Pointwise Convolution#

A pointwise convolution is a:

›1 × 1 convolution

It combines information across channels.

Therefore:

text
Ask AICopy
Depthwise:
spatial filtering

Pointwise:
channel mixing

8.4 Why MobileNet Is Efficient#

A standard convolution with:

text
Ask AICopy
K × K kernel
C_in input channels
C_out output channels

has:

›K × K × C_in × C_out

weights, ignoring bias.

A depthwise separable convolution has approximately:

›K × K × C_in

depthwise parameters plus:

›C_in × C_out

pointwise parameters.

Total:

›K × K × C_in + C_in × C_out

This can be substantially smaller than standard convolution when the kernel size and channel counts are large.

---

9. EfficientNet#

EfficientNet is a CNN family designed around systematic scaling of network size.

Instead of increasing only one architectural dimension, EfficientNet introduced compound scaling.

9.1 The Three Scaling Dimensions#

A CNN can be scaled in three major ways:

Architecture & Data Flow
Copy
Depth  -> more layers
Width  -> more channels
Resolution -> larger input images / feature maps

EfficientNet scales these dimensions together in a coordinated manner.

9.2 Compound Scaling#

Conceptually:

Architecture & Data Flow
Copy
Model size
   |
   +--> Depth
   |
   +--> Width
   |
   +--> Resolution

Instead of arbitrarily increasing one dimension, EfficientNet uses a compound scaling strategy to balance them.

9.3 EfficientNet-B0#

EfficientNet-B0 is the baseline architecture.

Larger variants are produced by scaling the baseline:

text
Ask AICopy
B0
B1
B2
B3
B4
B5
B6
B7

As the model variant increases, the network is systematically scaled to provide greater capacity at increased computational cost.

9.4 Efficient Building Blocks#

EfficientNet uses mobile inverted bottleneck convolution (MBConv) blocks.

These blocks are related to the efficient convolutional ideas used by MobileNet and commonly include:

Architecture & Data Flow
Copy
Expansion
   |
Depthwise Convolution
   |
Squeeze-and-Excitation
   |
Projection

Residual/shortcut connections are used where the tensor dimensions allow them.

9.5 Squeeze-and-Excitation#

The Squeeze-and-Excitation (SE) mechanism learns channel-wise importance.

Conceptually:

Architecture & Data Flow
Copy
Feature Maps
     |
     v
Global information
     |
     v
Channel weights
     |
     v
Reweight feature maps

This allows the network to emphasize more useful channels and reduce the contribution of less useful ones.

9.6 Importance of EfficientNet#

EfficientNet demonstrated that carefully balancing:

›Depth + Width + Resolution

can provide strong accuracy-efficiency trade-offs.

---

10. Depthwise Separable Convolution#

Depthwise separable convolution decomposes a standard convolution into two separate operations:

text
Ask AICopy
Depthwise Convolution
        +
Pointwise Convolution

This significantly reduces computation and parameter count compared with a standard convolution in many common settings.

10.1 Standard Convolution#

Suppose:

Mathematical Formulation
Copy
Kernel size = K × K
Input channels = C_in
Output channels = C_out

A standard convolution learns:

›K × K × C_in × C_out

weights.

The filter simultaneously considers:

text
Ask AICopy
spatial dimensions
+
all input channels

10.2 Depthwise Convolution#

Depthwise convolution separates the spatial operation by channel.

For:

›C_in input channels

the layer uses approximately:

›C_in

spatial kernels.

Parameter count:

›K × K × C_in

Each kernel operates on one input channel.

10.3 Pointwise Convolution#

After depthwise convolution, a 1 × 1 convolution mixes the channel information.

Parameter count:

›C_in × C_out

Therefore, the complete depthwise separable convolution has approximately:

text
Ask AICopy
K × K × C_in
+
C_in × C_out

parameters.

10.4 Parameter Comparison#

Standard convolution:

›K² × C_in × C_out

Depthwise separable convolution:

›K² × C_in + C_in × C_out

For example:

Mathematical Formulation
Copy
K = 3
C_in = 32
C_out = 64

Standard convolution:

Mathematical Formulation
Copy
3 × 3 × 32 × 64
= 18,432 parameters

Depthwise separable convolution:

Mathematical Formulation
Copy
3 × 3 × 32
+
32 × 64

= 288 + 2,048
= 2,336 parameters

This is a substantial reduction.

10.5 Computational Flow#

Architecture & Data Flow
Copy
Input
  |
  v
Depthwise Convolution
(one spatial filter per input channel)
  |
  v
Intermediate Feature Maps
  |
  v
1 × 1 Pointwise Convolution
(channel mixing)
  |
  v
Output Feature Maps

10.6 PyTorch Implementation#

PyTorch can implement depthwise convolution using:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
groups=in_channels

Example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

in_channels = 32
out_channels = 64

depthwise = nn.Conv2d(
    in_channels=in_channels,
    out_channels=in_channels,
    kernel_size=3,
    padding=1,
    groups=in_channels
)

pointwise = nn.Conv2d(
    in_channels=in_channels,
    out_channels=out_channels,
    kernel_size=1
)

X = torch.randn(8, 32, 32, 32)

x = depthwise(X)
output = pointwise(x)

print("Input shape :", X.shape)
print("Output shape:", output.shape)

The depthwise layer performs spatial filtering independently for each input channel.

The pointwise layer then combines those channels to produce the requested number of output channels.

10.7 Why It Is Important#

Depthwise separable convolution is especially useful when computational resources are limited.

It is a key building block in architectures such as:

›MobileNet

and is also used in other efficient CNN designs.

---

Architecture Comparison#

Quick Recap#

1. LeNet established the basic convolution → pooling → classification pattern.
2. AlexNet demonstrated the power of deeper CNNs, ReLU, GPUs, dropout, and max pooling at large scale.
3. VGG showed that a simple architecture built from repeated 3 × 3 convolutions could achieve strong performance by increasing depth.
4. GoogLeNet introduced the Inception approach to process information through multiple branches while controlling computational cost.
5. Inception uses parallel operations with different kernel sizes and pooling to capture information at multiple spatial scales.
6. ResNet uses residual connections to provide direct paths through deep networks and make optimization easier.
7. DenseNet connects layers densely and concatenates previous feature maps, encouraging feature reuse.
8. MobileNet uses depthwise separable convolution to reduce computation and parameter count for resource-constrained devices.
9. EfficientNet systematically scales depth, width, and input resolution together using compound scaling.
10. Depthwise separable convolution separates spatial filtering from channel mixing, making convolution much more computationally efficient.