Grade 12

Advanced

8 min

Lesson 10 of 12

Reinforcement Learning and Agents

Not Started

Understanding how agents learn to make decisions through rewards and penalties in an environment.

How does a computer program beat the world champion at Go or learn to play video games from scratch? It doesn't follow a pre-written script; it learns through the high-stakes game of trial, error, and rewards.

Learning Objectives

1.Define the components of the Markov Decision Process (State, Action, Reward).
2.Explain the critical trade-off between exploration and exploitation.
3.Describe how Q-learning uses the Bellman Equation to find an optimal policy.

The Foundation: Markov Decision Processes

At its core, Reinforcement Learning (RL) is about an agent interacting with an environment. This interaction is formalized as a Markov Decision Process (MDP). In this framework, the agent observes the current State ( $S_t$ ), performs an Action ( $A_t$ ), and receives a Reward ( $R_{t+1}$ ) while transitioning to a new state ( $S_{t+1}$ ). The 'Markov' property assumes that the future depends only on the current state, not the entire history of moves. The ultimate goal of the agent is to learn a Policy ( $\pi$ ), which is a strategy or mapping that tells the agent which action to take in any given state to maximize the cumulative reward over time, often called the Return ( $G_t$ ).

Imagine a robot on a $3 \times 3$ grid trying to reach a charging station. 1. **State ( $S$ ):** The robot's current coordinates, e.g., $(1, 1)$ . 2. **Action ( $A$ ):** Move Up, Down, Left, or Right. 3. **Reward ( $R$ ):** $+10$ for reaching the charger, $-1$ for hitting a wall, and $-0.1$ for every step taken (to encourage speed).

Quick Check

In an MDP, what do we call the strategy that dictates which action an agent should take in a specific state?

Answer

The Policy (denoted by the Greek letter $\pi$ ).

The Dilemma: Exploration vs. Exploitation

One of the toughest challenges in RL is the Exploration vs. Exploitation trade-off. Exploitation means the agent chooses the action it knows yields the highest reward based on current data. Exploration means the agent tries a random or unknown action to see if it leads to a better long-term outcome. If an agent only exploits, it might get stuck in a 'local optimum'—a decent path that isn't actually the best one. To balance this, we often use an ** $\epsilon$ -greedy strategy**, where the agent explores with probability $\epsilon$ and exploits with probability $1 - \epsilon$ .

Think of choosing a place for dinner: 1. Exploitation: You go to your favorite pizza place because you know it's an $8/10$ experience. 2. Exploration: You try the new sushi place down the street. It could be a $2/10$ (bad reward) or a $10/10$ (better than your current best). 3. The Trade-off: If you never explore, you'll never find the $10/10$ restaurant.

Quick Check

Why is pure exploitation risky for an AI agent?

Answer

The agent might miss out on much higher rewards because it stops searching for better paths once it finds a functional one.

Finding the Path: Q-Learning

How does the agent actually 'learn' the value of its actions? It uses Q-learning. The agent maintains a Q-table that stores Q-values,

Q(s, a)

, representing the expected total reward for taking action

a

in state

s

. We update these values using the Bellman Equation. The core idea is that the value of your current action is the reward you just got PLUS the discounted value of the best possible action in the next state:

Q(s, a) \leftarrow Q(s, a) + \alpha [R + \gamma \max_{a'} Q(s', a') - Q(s, a)]

Here,

\alpha

is the learning rate and

\gamma

(gamma) is the discount factor, which determines how much the agent cares about future rewards versus immediate ones.

Suppose an agent is in State A, takes an action, and receives a reward $R=5$ . It lands in State B. 1. Current $Q(A, action) = 10$ . 2. The best possible future value in State B is $\max Q(B, a') = 20$ . 3. Let learning rate $\alpha = 0.1$ and discount $\gamma = 0.9$ . 4. New Value: $10 + 0.1 [5 + (0.9 \times 20) - 10] = 10 + 0.1[13] = 11.3$ . The agent's 'knowledge' of that action just improved!

Key Takeaways

1The Markov Decision Process (MDP) consists of States, Actions, and Rewards.
2The Exploration vs. Exploitation trade-off balances using known rewards with searching for better ones.
3Q-learning uses the Bellman Equation to iteratively update the expected value of actions.
4The Discount Factor ( $\gamma$ ) determines the importance of future rewards relative to immediate ones.

Quick Check

Multiple Choice

Which component of an MDP represents the 'strategy' the agent follows?

Multiple Choice

In the $\epsilon$ -greedy strategy, if $\epsilon = 0.1$ , how often does the agent explore?

True/False

A higher discount factor ( $\gamma$ closer to 1) makes an agent more 'short-sighted,' focusing only on immediate rewards.

What's Next?

Review Tomorrow

In 24 hours, try to sketch the MDP loop (Agent -> Action -> Environment -> State/Reward) from memory and explain the difference between $\alpha$ and $\gamma$ .

Practice Activity

Research the 'Frozen Lake' environment in OpenAI Gym to see how Q-learning is implemented in Python code.

Browse

Browse

Reinforcement Learning and Agents

Learning Objectives

The Foundation: Markov Decision Processes

The Dilemma: Exploration vs. Exploitation

Finding the Path: Q-Learning

Key Takeaways

Quick Check

What's Next?

Reinforcement Learning and Agents

Learning Objectives

The Foundation: Markov Decision Processes

The Dilemma: Exploration vs. Exploitation

Finding the Path: Q-Learning

Key Takeaways

Quick Check

What's Next?