Grade 12

Advanced

7 min

Lesson 2 of 12

Linear Regression and Optimization

Not Started

Exploring the mathematical foundations of predictive modeling through linear regression and gradient descent.

How does a self-driving car predict exactly where a pedestrian will be in two seconds, or how does Netflix know which movie will be your next obsession? It all starts with a single line and the math of 'getting less wrong' over time.

Learning Objectives

1.Calculate a simple linear regression line using weights and biases.
2.Explain the Mean Squared Error (MSE) as a mathematical measure of model 'pain'.
3.Describe how Gradient Descent iteratively 'walks' down a cost curve to find the optimal solution.

The Linear Model: Mapping the World

At its core, Linear Regression is about finding the relationship between an input (independent variable

x

) and an output (dependent variable

y

). In Machine Learning, we express this as:

y = wx + b

Here,

w

is the Weight (the slope/strength of the relationship) and

b

is the Bias (the intercept). Our goal is to find the specific values of

w

and

b

that allow our line to pass as close as possible to all data points in a set. This line acts as a 'predictor'—if we know a new

x

, we can calculate the most likely

y

Imagine you want to predict a test score based on hours studied. 1. You have a weight

w = 5

(points per hour) and a bias

b = 50

(the base score). 2. If a student studies for

x = 4

hours, the prediction is:

y = (5 \times 4) + 50 = 70

3. If the actual score was 75, your model was off by 5 points. This difference is called the residual.

Quick Check

In the equation $y = wx + b$ , what happens to the predicted $y$ if we increase the bias $b$ ?

Think about the y-intercept in basic algebra.

Answer

The entire line shifts upward on the y-axis, increasing every prediction by the same amount.

The Cost Function: Measuring the Pain

To improve our model, we need a way to quantify how 'wrong' it is. We use a Cost Function, specifically Mean Squared Error (MSE). MSE calculates the average of the squares of the errors between actual values (

y_i

) and predicted values (

\\hat{y}_i

\begin{gathered}J(w, b) = \\frac{1}{n} \\sum_{i=1}^n (y_i - \\hat{y}_i)^2\end{gathered}

We square the errors for two reasons: it ensures all errors are positive, and it penalizes larger errors more heavily than smaller ones. A high cost means a poor model; a cost of zero means a perfect fit.

Let's calculate MSE for two data points: 1. Point A: Actual $y=10$ , Predicted $\\hat{y}=12$ . Error = $-2$ . Squared Error = $4$ . 2. Point B: Actual $y=20$ , Predicted $\\hat{y}=17$ . Error = $3$ . Squared Error = $9$ . 3. Sum of squares = $4 + 9 = 13$ . 4. $MSE = \\frac{13}{2} = 6.5$ .

Quick Check

Why do we square the errors in the MSE formula instead of just adding them up?

What happens if you add -5 and 5?

Answer

Squaring prevents positive and negative errors from canceling each other out and gives more weight to large outliers.

Gradient Descent: The Path to Perfection

How do we find the

w

and

b

that result in the lowest possible MSE? We use Gradient Descent. Imagine standing on a foggy mountain (the cost curve) and wanting to find the valley floor. You feel the slope under your feet and take a step in the steepest downward direction. In math, we calculate the partial derivative of the cost function with respect to

w

and

b

. We then update our parameters:

\begin{gathered}w_{new} = w_{old} - \\alpha \\frac{\\partial J}{\\partial w}\end{gathered}

The term

\\alpha

is the Learning Rate, which controls how big our steps are. If

\\alpha

is too large, we might overstep the valley; if it's too small, the model takes forever to learn.

Suppose your current weight $w$ is 10, your learning rate $\\alpha$ is 0.1, and the gradient (slope) at that point is 5. 1. The gradient is positive, meaning the 'mountain' goes up as $w$ increases. 2. To go down, we subtract: $w = 10 - (0.1 \times 5)$ . 3. New weight $w = 9.5$ . 4. We repeat this thousands of times until the gradient is nearly zero, meaning we've reached the bottom of the valley.

Key Takeaways

1Linear regression uses the formula $y = wx + b$ to model relationships between variables.
2The Mean Squared Error (MSE) formula is $J = \\frac{1}{n} \\sum (y - \\hat{y})^2$ , which measures model inaccuracy.
3Gradient Descent is an optimization algorithm that iteratively adjusts weights to minimize the cost function.
4The Learning Rate ( $\\alpha$ ) is a crucial hyperparameter that determines the size of the steps taken toward the minimum.

Quick Check

Multiple Choice

What does a gradient of zero indicate in the context of a cost function?

Multiple Choice

If your model's predictions are consistently overshooting the target and the error is getting larger, what should you adjust?

True/False

In simple linear regression, the Mean Squared Error (MSE) can sometimes be a negative number.

What's Next?

Review Tomorrow

In 24 hours, try to write down the MSE formula from memory and explain the 'mountain' analogy for gradient descent to a friend.

Practice Activity

Try to manually calculate one update step for a weight $w$ given a starting value of 2.0, a gradient of 0.4, and a learning rate of 0.01.

Browse

Browse

Linear Regression and Optimization

Learning Objectives

The Linear Model: Mapping the World

The Cost Function: Measuring the Pain

Gradient Descent: The Path to Perfection

Key Takeaways

Quick Check

What's Next?

Linear Regression and Optimization

Learning Objectives

The Linear Model: Mapping the World

The Cost Function: Measuring the Pain

Gradient Descent: The Path to Perfection

Key Takeaways

Quick Check

What's Next?