Neural Networks: Complete Notes (Beginner to Advanced)

---

1. Artificial Neuron#

An artificial neuron is the basic computational unit of a neural network. It is loosely inspired by a biological neuron: it receives multiple inputs, combines them, and produces an output.

How it works:

1. Each input x1, x2, ..., xn is multiplied by a corresponding weight w1, w2, ..., wn.
2. All weighted inputs are summed together.
3. A bias term b is added to this sum.
4. The result is passed through an activation function to produce the final output.

Mathematically:

$z = \sum_{i=1}^n w_i x_i + b = \mathbf{w}^T \mathbf{x} + b, \quad \text{output} = \sigma(z)$

Without an artificial neuron model: you would have no systematic way to combine multiple inputs into a single decision. You'd be forced to write manual if-else rules for every input combination, which does not scale and cannot learn from data.

With an artificial neuron: the weights and bias are learned automatically from data, so the neuron adjusts itself to find the right combination of inputs that leads to correct predictions.

Code example (single neuron in Python using NumPy):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def artificial_neuron(inputs, weights, bias, activation_fn):
    z = np.dot(inputs, weights) + bias
    return activation_fn(z)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

inputs = np.array([0.5, 0.8, 0.2])
weights = np.array([0.4, 0.7, 0.1])
bias = 0.1

output = artificial_neuron(inputs, weights, bias, sigmoid)
print("Neuron output:", output)

---

2. Perceptron#

The perceptron is the earliest and simplest type of artificial neuron, introduced by Frank Rosenblatt in 1958. It is a binary linear classifier: it takes several inputs and produces a single binary output (0 or 1).

Perceptron formula:

$z = \mathbf{w}^T \mathbf{x} + b, \quad y = \begin{cases} 1 & \text{if } z > 0 \\ 0 & \text{otherwise} \end{cases}$

The perceptron uses a step function as its activation function, meaning it outputs a hard 0 or 1 rather than a smooth probability.

Learning rule (Perceptron Learning Algorithm):

Mathematical Formulation
Copy
w_new = w_old + learning_rate * (y_true - y_predicted) * x
b_new = b_old + learning_rate * (y_true - y_predicted)

Limitation: A single perceptron can only solve linearly separable problems. It cannot solve problems like XOR, because no single straight line can separate the classes.

Without stacking perceptrons: you are limited to linear decision boundaries only, so complex real-world patterns (images, speech, non-linear relationships) cannot be modeled.

With multiple perceptrons stacked into layers (Multi-Layer Perceptron): the network can approximate any non-linear function, which is why modern neural networks use many neurons in multiple layers rather than a single perceptron.

Code example (Perceptron from scratch):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

class Perceptron:
    def __init__(self, num_inputs, learning_rate=0.1, epochs=50):
        self.weights = np.zeros(num_inputs)
        self.bias = 0.0
        self.lr = learning_rate
        self.epochs = epochs

    def step_function(self, z):
        return 1 if z > 0 else 0

    def predict(self, x):
        z = np.dot(x, self.weights) + self.bias
        return self.step_function(z)

    def train(self, X, y):
        for _ in range(self.epochs):
            for xi, target in zip(X, y):
                prediction = self.predict(xi)
                error = target - prediction
                self.weights += self.lr * error * xi
                self.bias += self.lr * error

# Example: AND gate
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0,0,0,1])

p = Perceptron(num_inputs=2)
p.train(X, y)

for xi in X:
    print(xi, "->", p.predict(xi))

---

3. Neural Network Architecture#

A neural network's architecture refers to how neurons are organized and connected. The most common architecture is a feedforward network, where information flows in one direction: from input, through hidden layers, to output.

Core architectural elements:

• Number of layers (depth)
• Number of neurons per layer (width)
• Connections between neurons (fully connected, convolutional, recurrent, etc.)
• Activation functions used at each layer
• Direction of information flow (feedforward vs recurrent)

Without a well-designed architecture: the network may underfit (too small to capture the pattern) or overfit (too large, memorizing training data instead of generalizing).

With the right architecture: the network has enough capacity to learn the underlying pattern in the data while still generalizing well to new, unseen data.

A simple architecture diagram (text form):

Input Layer -> Hidden Layer 1 -> Hidden Layer 2 -> ... -> Output Layer

---

4. Layers#

A layer is a collection of neurons that process inputs at the same stage of the network. Data passes sequentially through layers.

4.1 Input Layer#

The input layer receives the raw data and passes it into the network. It does not perform any computation; it simply represents the features of your dataset.

• Number of neurons in the input layer = number of features in your data.
• Example: if you have an image of 28x28 pixels, the input layer has 784 neurons (28*28) when flattened.

Without a correctly sized input layer: the network cannot accept your data at all, or it will silently misinterpret feature dimensions.

4.2 Hidden Layers#

Hidden layers sit between the input and output layers. They perform the actual transformation and feature extraction, learning increasingly abstract representations of the data as depth increases.

• A network with one or more hidden layers is called a Multi-Layer Perceptron (MLP) or Deep Neural Network if it has many hidden layers.
• Each hidden layer applies a linear transformation followed by a non-linear activation function.

Without hidden layers: the network reduces to a simple linear model (like linear/logistic regression) and cannot learn non-linear patterns.

With hidden layers: the network can approximate complex, non-linear functions (this is guaranteed by the Universal Approximation Theorem).

4.3 Output Layer#

The output layer produces the final prediction of the network. Its structure depends on the task:

Code example (defining layers using Keras/TensorFlow):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Input(shape=(784,)),              # Input layer
    layers.Dense(128, activation='relu'),      # Hidden layer 1
    layers.Dense(64, activation='relu'),       # Hidden layer 2
    layers.Dense(10, activation='softmax')     # Output layer (10 classes)
])

model.summary()

---

5. Weights#

Weights are learnable parameters that determine the strength and direction (positive or negative) of the connection between two neurons. Every connection between neurons has its own weight.

• A large positive weight means that input strongly increases the neuron's activation.
• A large negative weight means that input strongly decreases the neuron's activation.
• A weight close to zero means that input has little influence.

Without weights (or with all weights fixed/random and untrained): the network cannot learn anything meaningful; its output would just be a fixed, arbitrary transformation of the input, unrelated to the actual data pattern.

With trained weights: the network encodes the relationship between inputs and outputs, learned directly from the training data through backpropagation.

Weight initialization matters:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import tensorflow as tf

layer = tf.keras.layers.Dense(
    64,
    activation='relu',
    kernel_initializer='he_normal'  # good for ReLU-based networks
)

Common initialization strategies:

• Zero initialization: Bad. All neurons learn the same thing (symmetry problem).
• Random initialization: Basic, but can cause vanishing/exploding gradients if not scaled properly.
• Xavier/Glorot initialization: Good for Sigmoid/Tanh activations.
• He initialization: Good for ReLU-based activations.

---

6. Bias#

Bias is an additional learnable parameter added to the weighted sum of inputs before applying the activation function. It allows the neuron to shift its activation function left or right, independent of the input values.

z = (w1*x1 + w2*x2 + ... + wn*xn) + b

Without bias: the neuron's output is forced to pass through the origin, meaning it can only represent functions that pass through zero when all inputs are zero. This severely limits what the network can learn.

With bias: the neuron can represent any linear shift, giving the network much more flexibility to fit the data correctly.

Analogy: think of bias like the intercept b in the line equation y = mx + b. Without it, every line would be forced to pass through the origin (0,0), which is rarely what the actual data looks like.

---

7. Parameters#

Parameters are the values a neural network learns during training. They consist of:

• Weights (connection strengths)
• Biases (shift values)

Total number of parameters in a fully connected (dense) layer:

Mathematical Formulation
Copy
parameters = (number_of_inputs * number_of_neurons) + number_of_neurons
             \_______weights________/               \____biases____/

Example calculation:

For a Dense layer with 784 inputs and 128 neurons:

Mathematical Formulation
Copy
weights = 784 * 128 = 100,352
biases = 128
total parameters = 100,480

Code example (counting parameters):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Input(shape=(784,)),
    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

model.summary()  # shows param count per layer and total

Without tracking parameter count: you risk building a model too large for your dataset (overfitting, slow training) or too small (underfitting).

With awareness of parameter count: you can balance model capacity against dataset size and compute budget.

---

8. Linear Transformation#

Before activation, every layer performs a linear transformation on its inputs. This is the weighted sum plus bias operation, expressed in matrix form for an entire layer:

Z = X . W + b

Where:

• X is the input matrix (batch_size x input_features)
• W is the weight matrix (input_features x output_neurons)
• b is the bias vector (output_neurons,)
• Z is the pre-activation output (batch_size x output_neurons)

Without the activation function after this step: stacking many linear transformations is mathematically equivalent to a single linear transformation. No matter how many layers you add, the entire network collapses into one linear function, which cannot model non-linear relationships (this is why activation functions are essential, covered next).

With a linear transformation as the foundation: each layer performs a well-defined, differentiable operation that can be optimized using gradient-based methods like backpropagation.

Code example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

X = np.array([[1.0, 2.0, 3.0]])       # 1 sample, 3 features
W = np.random.randn(3, 4)              # 3 inputs -> 4 neurons
b = np.zeros(4)

Z = np.dot(X, W) + b
print("Linear transformation output (Z):", Z)

---

9. Activation Functions#

Activation functions introduce non-linearity into the network. Without them, a neural network of any depth behaves like a single linear layer, no matter how many layers you stack.

Without activation functions:

• Error: The network can only learn linear relationships.
• Example: it would fail to distinguish an XOR pattern, or model any curved decision boundary, or learn hierarchical features in images.

With activation functions:

• The network can approximate any continuous function (Universal Approximation Theorem), enabling it to learn complex patterns like image recognition, language understanding, etc.

Below are the most common activation functions, each with formula, characteristics, and code.

9.1 Sigmoid#

sigmoid(z) = 1 / (1 + e^(-z))

• Output range: (0, 1)
• Historically used in output layers for binary classification and in hidden layers of early networks.
• Problem: Suffers from the vanishing gradient problem for very large or very small z, since the gradient approaches zero at the extremes, slowing down learning in deep networks.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

print(sigmoid(np.array([-2, 0, 2])))

Without sigmoid: you'd lack a smooth way to squash outputs into a (0,1) probability range for binary classification. With sigmoid: the output can be directly interpreted as a probability, useful for binary classification output layers.

9.2 Tanh (Hyperbolic Tangent)#

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

• Output range: (-1, 1)
• Zero-centered, which usually helps optimization converge faster than sigmoid.
• Still suffers from vanishing gradients at extreme values, though less severely than sigmoid.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def tanh(z):
    return np.tanh(z)

print(tanh(np.array([-2, 0, 2])))

Without zero-centering (as in sigmoid): gradients during backpropagation tend to move consistently in one direction, causing zig-zagging updates and slower convergence. With tanh's zero-centered output: gradients can push weights in both positive and negative directions more evenly, generally speeding up convergence relative to sigmoid.

9.3 ReLU (Rectified Linear Unit)#

ReLU(z) = max(0, z)

• Output range: [0, infinity)
• The most widely used activation function in modern deep learning due to its simplicity and efficiency.
• Problem: "Dying ReLU" — neurons can get stuck outputting 0 permanently if a large negative gradient updates the weights such that the neuron never activates again.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def relu(z):
    return np.maximum(0, z)

print(relu(np.array([-2, 0, 2])))

Without ReLU (using sigmoid/tanh in deep networks): deep networks train very slowly or fail to converge due to vanishing gradients across many layers. With ReLU: gradients do not vanish for positive inputs, allowing much deeper networks to be trained efficiently. It's also computationally cheap (just a max operation).

9.4 Leaky ReLU#

LeakyReLU(z) = z if z > 0 else alpha * z   (alpha is a small constant, e.g., 0.01)

• Fixes the dying ReLU problem by allowing a small, non-zero gradient when the unit is not active.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)

print(leaky_relu(np.array([-2, 0, 2])))

Without Leaky ReLU (plain ReLU): neurons with consistently negative input get zero gradient and stop learning entirely ("dead neurons"). With Leaky ReLU: even negative inputs produce a small gradient, so neurons can recover and continue learning.

9.5 ELU (Exponential Linear Unit)#

ELU(z) = z if z > 0 else alpha * (e^z - 1)

• Smooths out the negative side using an exponential curve instead of a straight line (unlike Leaky ReLU).
• Tends to push mean activations closer to zero, which can speed up learning.
• Computationally more expensive than ReLU due to the exponential term.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def elu(z, alpha=1.0):
    return np.where(z > 0, z, alpha * (np.exp(z) - 1))

print(elu(np.array([-2, 0, 2])))

Without ELU (using ReLU): the mean activation output tends to stay positive and shifted away from zero, which can slow down convergence. With ELU: activations are pushed closer to a zero mean, similar to batch normalization's effect, often leading to faster convergence.

9.6 GELU (Gaussian Error Linear Unit)#

GELU(z) = z * Phi(z)

Where Phi(z) is the cumulative distribution function (CDF) of the standard normal distribution. A common practical approximation is:

GELU(z) ≈ 0.5 * z * (1 + tanh(sqrt(2/pi) * (z + 0.044715 * z^3)))

• Used in modern transformer architectures like BERT and GPT.
• Smoothly weights inputs by their value rather than a hard cutoff like ReLU, effectively behaving as a smoother version of ReLU with probabilistic weighting.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def gelu(z):
    return 0.5 * z * (1 + np.tanh(np.sqrt(2 / np.pi) * (z + 0.044715 * np.power(z, 3))))

print(gelu(np.array([-2, 0, 2])))

Without GELU (using plain ReLU in transformers): models tend to perform slightly worse in practice on language tasks; the hard cutoff of ReLU is less smooth for the kind of optimization landscapes transformers create. With GELU: transformer models like BERT and GPT achieve smoother gradients and empirically better performance, which is why it became the standard choice for these architectures.

9.7 Softmax#

softmax(z_i) = e^(z_i) / sum(e^(z_j) for all j)

• Converts a vector of raw scores (logits) into a probability distribution where all values sum to 1.
• Used exclusively in the output layer for multi-class classification problems.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def softmax(z):
    exp_z = np.exp(z - np.max(z))  # subtract max for numerical stability
    return exp_z / np.sum(exp_z)

logits = np.array([2.0, 1.0, 0.1])
print(softmax(logits))

Without softmax: raw output scores (logits) from a multi-class network are not interpretable as probabilities, and you can't directly compare class likelihoods or apply cross-entropy loss correctly. With softmax: the output layer produces a clean probability distribution over classes, letting you pick the class with the highest probability and train the network with categorical cross-entropy loss.

---

Summary Table: Choosing an Activation Function#

---

Quick Recap (Beginner to Advanced Flow)#

1. An artificial neuron combines inputs using weights and bias, then applies an activation function.
2. A perceptron is the simplest neuron model, limited to linearly separable problems.
3. Neurons are organized into a neural network architecture made of layers: input, hidden, and output.
4. Weights and bias are the learnable parameters, together called parameters, that the network adjusts during training.
5. Each layer performs a linear transformation (Z = XW + b) before applying a non-linear activation function.
6. Activation functions like Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, GELU, and Softmax give the network the ability to learn complex, non-linear patterns, each with trade-offs in gradient behavior and computational cost.