Grade 12

Advanced

10 min

Lesson 6 of 12

Deep Learning and Backpropagation

Not Started

Understanding how multi-layer neural networks learn through the process of backpropagation and the chain rule.

How does a computer 'learn' from its mistakes? Imagine a blindfolded hiker trying to find the lowest point in a valley by only feeling the slope of the ground under their feet—this is exactly how the world's most advanced AI learns.

Learning Objectives

1.Describe the architecture of a Multi-Layer Perceptron (MLP) including input, hidden, and output layers.
2.Apply the mathematical chain rule to calculate how specific weights contribute to total model error.
3.Analyze how the learning rate parameter influences the speed and stability of model convergence.

The Architecture: The Multi-Layer Perceptron

A Multi-Layer Perceptron (MLP) is the foundational architecture of deep learning. It consists of an Input Layer, one or more Hidden Layers, and an Output Layer. Each layer contains neurons (nodes) connected by weights ( $w$ ). When data flows forward, each neuron calculates a weighted sum of its inputs, adds a bias ( $b$ ), and passes the result through an activation function like ReLU or Sigmoid. This process, called Forward Propagation, results in a prediction. However, the initial prediction is usually wrong. To improve, the network must calculate a Loss Function $L$ , which measures the distance between the prediction $\hat{y}$ and the actual target $y$ .

Quick Check

In an MLP, what three components are combined at a neuron before being passed to an activation function?

Think of the linear equation

y = mx + b

Answer

The weighted sum of inputs, the weights themselves, and a bias term.

The Engine: The Chain Rule

To reduce the error, we need to know how much each individual weight contributed to the total loss. This is where Backpropagation begins. We use the Chain Rule from calculus to find the gradient of the loss function with respect to each weight. If we have a composite function

z = f(g(x))

, the derivative of

z

with respect to

x

is:

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial g} \cdot \frac{\partial g}{\partial x}

In a neural network, we 'chain' these derivatives backward from the output layer through the hidden layers to the input. This tells us the sensitivity of the loss to every single parameter in the system.

Suppose we have a simple network where the loss $L$ depends on an output $a$ , and $a$ depends on a weight $w$ . 1. Let $L = (a - y)^2$ and $a = w \cdot x$ . 2. To find how $L$ changes with $w$ , we calculate: $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial w}$ . 3. $\frac{\partial L}{\partial a} = 2(a - y)$ and $\frac{\partial a}{\partial w} = x$ . 4. Therefore, $\frac{\partial L}{\partial w} = 2(a - y) \cdot x$ .

The Adjustment: Gradient Descent and Learning Rates

Once we have the gradient (the 'slope'), we update the weights to move 'downhill' toward the minimum loss. This is Gradient Descent. The size of the step we take is determined by the Learning Rate (denoted by the Greek letter

\eta

, eta). The update rule is:

w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w}

\eta

is too high, the model might 'overshoot' the minimum and diverge. If

\eta

is too low, the model will take a very long time to learn, potentially getting stuck in a local minimum.

Imagine your gradient $\frac{\partial L}{\partial w}$ is $10$ and your current weight is $5$ . 1. If $\eta = 0.1$ , your new weight is $5 - (0.1 \cdot 10) = 4$ . A steady, controlled step. 2. If $\eta = 1.0$ , your new weight is $5 - (1.0 \cdot 10) = -5$ . You have jumped far across the valley floor, potentially missing the bottom entirely.

Quick Check

Why do we subtract the gradient from the weight instead of adding it?

Answer

We subtract because the gradient points in the direction of the steepest increase; to minimize loss, we must move in the opposite direction (downhill).

Consider a weight $w_1$ in the first layer. The loss $L$ is calculated at the end of the second layer. To update $w_1$ , you must chain through the entire network: 1. Calculate $\frac{\partial L}{\partial ext{Output}}$ . 2. Multiply by $\frac{\partial ext{Output}}{\partial ext{Hidden}}$ . 3. Multiply by $\frac{\partial ext{Hidden}}{\partial w_1}$ . 4. This sequence $\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial out} \cdot \frac{\partial out}{\partial hid} \cdot \frac{\partial hid}{\partial w_1}$ allows information about the error to reach the very beginning of the network.

Key Takeaways

1Backpropagation is the process of calculating gradients from the output layer back to the input layer using the chain rule.
2The gradient $\frac{\partial L}{\partial w}$ represents how much a change in weight $w$ affects the total error $L$ .
3The learning rate $\eta$ is a hyperparameter that controls the step size during weight updates; balancing it is crucial for convergence.
4MLPs learn by iteratively performing forward passes to get errors and backward passes to update weights.

Quick Check

Multiple Choice

Which mathematical principle allows backpropagation to calculate gradients across multiple layers?

Multiple Choice

What happens if the learning rate $\eta$ is set too high?

True/False

In backpropagation, weights are updated before the loss is calculated.

What's Next?

Review Tomorrow

In 24 hours, try to write down the weight update formula $w_{new} = w_{old} - \eta \cdot \nabla L$ from memory and explain what each symbol represents.

Practice Activity

Sketch a 3-layer MLP (Input, 1 Hidden, Output) and draw the arrows for both the forward pass (predictions) and the backward pass (gradients).

Browse

Browse

Deep Learning and Backpropagation

Learning Objectives

The Architecture: The Multi-Layer Perceptron

The Engine: The Chain Rule

The Adjustment: Gradient Descent and Learning Rates

Key Takeaways

Quick Check

What's Next?

Deep Learning and Backpropagation

Learning Objectives

The Architecture: The Multi-Layer Perceptron

The Engine: The Chain Rule

The Adjustment: Gradient Descent and Learning Rates

Key Takeaways

Quick Check

What's Next?