11 Learning by Doing

Reinforcement Learning, Reward, and the Agent That Teaches Itself

CSC5350 · Artificial Intelligence

11.1 Opening Narrative

11.1.1 180 Years of Practice Per Day

In June 2018, OpenAI announced that its reinforcement learning system, OpenAI Five, had defeated a team of former professional Dota 2 players. By August it had beaten professionals in a full five-versus-five match. By April 2019 it had defeated the reigning world champions, OG, at the Dota 2 International — the most prestigious tournament in the game.

Dota 2 is, by almost any measure, one of the most complex games ever created. The state space — all the possible configurations of units, items, abilities, cooldowns, vision, and map states — dwarfs chess by many orders of magnitude. The game plays out over thirty to sixty minutes in real time, requiring strategic decisions at a horizon of thousands of frames per second. Human professionals spend years mastering it.

OpenAI Five learned to play Dota 2 in approximately ten months. It was not shown recordings of human gameplay. It was not given strategic heuristics from professional players. It was given one thing: a reward signal. Win the game, receive a positive reward. Lose, receive a negative reward. Everything else — the subtleties of itemization, the timing of team fights, the psychological pressure of a comeback — it discovered by playing against itself.

At peak training, OpenAI Five was running 180 years of simulated Dota 2 per day across its compute cluster.

The result was a system of extraordinary capability and revealing fragility. When the rules changed slightly, OpenAI Five needed to be retrained almost from scratch. When facing playing styles it had not encountered during training, it made mistakes that experienced human players would not. Its strategies were sometimes superhuman; sometimes they were bizarre, reflecting optimization for the reward signal in ways no human coach would have endorsed.

OpenAI Five illustrated both the promise and the characteristic failure mode of reinforcement learning: a system that becomes extraordinarily capable at optimizing whatever it was trained to optimize, within the specific distribution it was trained on — and that can behave unpredictably when either the objective or the distribution changes.

“A reinforcement learning agent does not learn to do the right thing. It learns to maximize its reward. Whether those two are the same depends entirely on how carefully the reward was designed — and on how much of the real world the training environment captured.”

11.2 Learning Objectives

After completing this chapter, you will be able to:

Describe the reinforcement learning framework — agent, environment, state, action, reward, policy, and value function — and distinguish it from supervised and unsupervised learning.
Formalize RL problems as Markov Decision Processes and connect the Bellman equation to Q-learning.
Explain the Deep Q-Network architecture and describe the roles of experience replay and target networks.
Distinguish value-based from policy gradient methods, and explain the actor-critic synthesis.
Analyze the exploration-exploitation tradeoff and compare ε-greedy, UCB, and curiosity-driven strategies.
Explain reward shaping, its acceleration benefits, and the alignment risks it introduces.
Describe how AlphaGo Zero used self-play to surpass all human knowledge without human game data.
Identify real-world RL applications — robotics, recommendation systems, RLHF — and their ethical dimensions.
Build the IAAIS Decision Agent: a Q-learning component integrated with your system’s planning module.

11.3 Key Terminology

Term	Plain-Language Definition
Agent	The learner and decision-maker in RL. Perceives its environment through sensors, selects actions, and updates its behavior based on received rewards.
Environment	Everything the agent interacts with but does not directly control. Receives the agent’s actions and returns new states and rewards.
State (s)	A description of the environment at a given moment, sufficient to predict future states and rewards under the Markov property.
Action (a)	A choice made by the agent. Actions may be discrete (move left/right) or continuous (apply a specific force).
Reward (r)	A scalar signal received after each action, indicating how desirable that transition was. The agent’s objective is to maximize cumulative future reward.
Policy (π)	A mapping from states to actions (or action probability distributions). The agent’s behavior strategy — what it does in each situation.
Value Function V(s)	The expected cumulative discounted reward the agent will collect from state s, following its current policy. What a state is “worth” in the long run.
Action-Value Function Q(s,a)	The expected cumulative discounted reward from taking action a in state s, then following the policy thereafter. The value of a specific decision.
Bellman Equation	The recursive consistency condition for value functions: the value of a state equals the immediate reward plus the discounted value of the next state. Foundation of Q-learning.
Discount Factor (γ)	A value in [0,1) weighting future rewards. Low γ is myopic; high γ is farsighted. γ=0.99 means a reward ten steps away is worth about 90% of an immediate reward.
Return (G_t)	The cumulative discounted reward from time t onward. Reinforcement learning aims to maximize expected return, not just immediate reward.
Q-Learning	A model-free RL algorithm that learns the optimal action-value function by iteratively applying the Bellman optimality update after each observed transition.
Deep Q-Network (DQN)	A neural network that approximates the Q-function from raw state inputs — enabling Q-learning at scales where a lookup table is impossible.
Experience Replay	Storing past transitions in a buffer and sampling randomly during training, breaking temporal correlations that destabilize neural network learning.
Target Network	A periodically updated copy of the Q-network used to compute stable training targets, preventing the feedback loop that causes Q-learning divergence.
Policy Gradient	A family of algorithms that directly optimize policy parameters by gradient ascent on expected return, without first learning a value function.
Advantage Function A(s,a)	Q(s,a) − V(s): how much better action a is compared to the agent’s average action in state s. Reduces variance in policy gradient updates.
Actor-Critic	An architecture combining a policy network (actor) with a value function network (critic). The actor selects actions; the critic evaluates them, providing lower-variance gradient estimates.
PPO	Proximal Policy Optimization. A state-of-the-art actor-critic algorithm that constrains policy updates to prevent destabilizing large changes. Powers OpenAI Five and ChatGPT alignment.
Exploration vs. Exploitation	The fundamental RL tradeoff: try new actions to discover better strategies (explore), or repeat known good actions (exploit)? Too little exploration misses better strategies; too much wastes reward.
ε-Greedy	A simple exploration strategy: take the greedy action with probability 1−ε and a random action with probability ε. Effective with carefully scheduled ε decay.
Reward Shaping	Adding supplementary reward signals to guide learning when the true reward is sparse. Accelerates learning but risks teaching the agent to pursue the shaped reward instead of the true goal.
Sparse Reward	A reward signal that is nonzero only rarely — for example, only upon winning or losing. Makes exploration extremely difficult; the agent rarely receives the feedback it needs.

11.4 Section 1 — Learning Without Labels

Supervised learning requires labeled examples — pairs of (input, correct output) produced by human experts. Unsupervised learning requires data but no labels. Reinforcement learning requires neither: the agent learns from the consequences of its own actions.

This is both RL’s most powerful property and its most fundamental challenge. An agent that does not need labeled data can be applied to problems where labeling is impossible, too expensive, or would encode only human-level solutions when the goal is superhuman performance. But an agent that learns only from consequences must discover what is worth doing through trial and error — potentially making millions of mistakes before finding a good strategy.

The key difference from supervised learning is temporal. A supervised learner sees one example, makes one prediction, receives one correction. An RL agent takes a sequence of actions, receives a sequence of rewards, and must figure out which actions caused which rewards — the credit assignment problem. If an agent wins a game of Dota 2, which of the thousands of actions it took over 40 minutes were responsible? Which were neutral? Which were mistakes it got away with? Solving credit assignment across time is the core technical challenge of reinforcement learning.

11.5 Section 2 — The RL Framework: MDPs and the Bellman Equation

11.5.1 Markov Decision Processes

Chapter 4 introduced Markov Decision Processes as a planning framework. RL uses the same formalism — but instead of knowing the transition and reward functions, the agent must learn them from interaction.

An MDP has five components: a state space S, an action space A, a transition function T(s, a, s’) expressing the probability of moving to state s’ after taking action a in state s, a reward function R(s, a) giving the expected immediate reward, and a discount factor γ.

The Markov property — that the current state contains all information needed to predict the future — is the foundational assumption. Given where you are, how you got there is irrelevant for optimal decision-making. This is a simplification, but a productive one.

11.5.2 The Bellman Equation

The central mathematical insight of RL is that value functions satisfy a recursive consistency condition. The value of a state equals the expected immediate reward plus the discounted value of the state that follows:

V*(s) = max_a [ R(s,a) + γ Σ_{s’} T(s,a,s’) V*(s’) ]

This is the Bellman optimality equation. Q-learning applies it as a learning update: after observing the transition (s, a, r, s’), adjust the estimate of Q(s, a) toward the target r + γ max_{a’} Q(s’, a’):

Q(s, a) ← Q(s, a) + α [ r + γ max_{a’} Q(s’, a’) − Q(s, a) ]

The bracketed term is the TD error — the surprise between what the agent expected and what it observed. Over many updates, Q-learning provably converges to the optimal action-value function, provided every state-action pair is visited sufficiently often.

# The Q-learning update: the heart of tabular reinforcement learning
# This single line embodies the Bellman optimality principle.

def q_learning_update(Q, state, action, reward, next_state,
                      alpha=0.1, gamma=0.99, done=False):
    """
    Apply one Bellman-based Q-learning update.
    Q:         dict mapping (state, action) → estimated value
    alpha:     learning rate — how fast to update
    gamma:     discount factor — how much future rewards matter
    """
    current_estimate = Q[(state, action)]

    if done:
        target = reward                          # Terminal: no future
    else:
        best_next = max(Q[(next_state, a)] for a in range(4))
        target = reward + gamma * best_next      # Bellman target

    td_error = target - current_estimate
    Q[(state, action)] += alpha * td_error       # Move toward target

    return td_error

Expected output (when called in training loop):

Cycle  100: avg TD error = 1.823
Cycle  500: avg TD error = 0.412
Cycle 1000: avg TD error = 0.089
Cycle 2000: avg TD error = 0.021   ← converging toward optimal Q*

11.5.3 The Exploration-Exploitation Tradeoff

Q-learning converges in theory — but convergence requires visiting every state-action pair sufficiently often. This demands exploration: sometimes the agent must try actions that appear suboptimal, because it cannot know they are suboptimal without trying them.

The canonical exploration strategy is ε-greedy: with probability ε, take a random action; with probability 1−ε, take the action with the highest Q-value. As training progresses, ε is decayed — the agent gradually shifts from exploring to exploiting what it has learned.

This tradeoff has no perfect solution. Every exploration strategy is a bet on what kind of environment the agent lives in. UCB (Upper Confidence Bound) exploration is optimistic about unexplored actions; Thompson Sampling maintains probability distributions over Q-values; curiosity-driven exploration rewards the agent for encountering novel states. The right choice depends on how sparse the rewards are and how large the state space is.

11.6 Section 3 — Deep Q-Networks: Q-Learning at Scale

Tabular Q-learning maintains one value per (state, action) pair. For small environments with dozens of states, this is trivial. For an Atari game where the state is a 210×160×3 pixel frame, the number of possible states is astronomically large. A lookup table is impossible.

Deep Q-Networks (DQN), introduced by DeepMind in 2013, solve this by replacing the Q-table with a neural network. The network takes the current state as input and outputs Q-values for all possible actions simultaneously. A convolutional network processes the raw pixels; fully connected layers produce the Q-value estimates.

11.6.1 Two Stabilization Innovations

Training a neural network against Q-learning targets is inherently unstable — the same network is used to generate both the predictions and the targets, creating a feedback loop that can cause divergence. DQN introduced two innovations that stabilized training:

Experience replay stores each transition (s, a, r, s’, done) in a buffer as it is collected. During training, the network samples random mini-batches from this buffer rather than learning from sequential experience. This breaks the temporal correlations that violate gradient descent’s i.i.d. assumption — the same transitions are replayed many times, and temporally adjacent experiences are decorrelated.

Target networks maintain a separate copy of the Q-network whose weights are frozen for C steps. The frozen copy computes the training targets; the main network is updated each step. Every C steps, the target network’s weights are replaced with the main network’s current weights. This prevents the destabilizing feedback loop of chasing a moving target.

With these two innovations, DQN trained from raw pixels to achieve superhuman performance on 49 Atari games — a result that would have been impossible with tabular methods and stunning even to the researchers who produced it.

Atari Game	Human Score	DQN Score	Human-Normalized
Breakout	31.8	401.2	1261%
Space Invaders	1652	1976	120%
Pong	9.3	18.9	203%
Seaquest	20182	5286	26%
Montezuma’s Revenge	4753	0	0%

The Montezuma’s Revenge result — zero points — foreshadowed the next challenge. Atari games with dense, frequent rewards were tractable. Games requiring long-horizon exploration before any reward was collected were not. Sparse rewards remained the unsolved frontier.

11.7 Section 4 — Policy Gradients and Actor-Critic Methods

11.7.1 From Value Functions to Policies

DQN is a value-based method: it learns a value function and derives a policy from it by acting greedily. This works well for discrete action spaces — the greedy action is simply the one with the highest Q-value. But for continuous action spaces — a robot arm applying a torque in [−5, 5] Nm — there is no discrete set of actions to maximize over.

Policy gradient methods take a different approach: directly parameterize the policy as a neural network and optimize its parameters by gradient ascent on expected return.

The policy gradient theorem establishes that the gradient of expected return with respect to policy parameters θ is:

**∇_θ J(θ) = E_π [ ∇_θ log π_θ(a|s) · G_t ]**

Intuitively: update parameters in the direction of actions that led to high return, scaled by the log-probability of those actions. This is the REINFORCE algorithm — the simplest policy gradient method — and it works, but with high variance: a single trajectory’s return fluctuates dramatically, producing noisy gradient estimates that slow learning.

11.7.2 The Actor-Critic Architecture

Actor-critic methods reduce variance by replacing the raw return G_t with the advantage function A(s, a) = Q(s, a) − V(s). The advantage measures how much better action a was compared to what the agent would have done on average in state s. Positive advantage: the action was better than expected; update toward it. Negative advantage: worse than expected; update away from it.

The architecture has two networks: an actor that outputs the policy distribution, and a critic that estimates the state value function used to compute advantages. The critic provides a baseline that removes the variance attributable to the inherent value of the state, leaving only the signal about whether the specific action was good or bad.

PPO (Proximal Policy Optimization) extends actor-critic with a constraint that prevents any single update from changing the policy too dramatically. The clipped surrogate objective ensures that even if the advantage estimate is large, the policy parameters move only within a trust region. This makes training stable across a wide range of hyperparameter settings — which is why PPO became the de facto standard for production deep RL, powering OpenAI Five, robotics systems at DeepMind, and the RLHF alignment step of ChatGPT and Claude.

11.8 Section 5 — AlphaGo Zero: Beyond Human Knowledge

The most dramatic demonstration of what RL can achieve beyond human knowledge is AlphaGo Zero — the system that mastered Go without any human game data.

The original AlphaGo (2016) used supervised pre-training on 160,000 recorded human games before RL fine-tuning. AlphaGo Zero (2017) removed the human data entirely. It started from random play and trained exclusively through self-play: at each step, the current network played games against itself, and both the policy and value heads were trained on the outcomes of those games.

The training loop was elegantly simple. The current best network generates games through MCTS-guided self-play. A new network is trained on those games. If the new network beats the current best in a head-to-head evaluation, it becomes the new current best. Repeat.

What emerged was remarkable. After just three days of training on four TPUs, AlphaGo Zero surpassed the version of AlphaGo that had defeated world champion Lee Sedol — a system that had taken months to train with human data. After 40 days, it surpassed every previous AlphaGo version, reaching a level of play that no human has ever achieved.

More striking still: the strategies AlphaGo Zero developed were genuinely novel. It rediscovered classic joseki (established opening sequences) that human players had refined over centuries — and then abandoned some of them in favor of patterns that human players had considered weak or unconventional. It had discovered something humans had missed.

Days of Training	Performance Level
0	Random play
1	Surpassed AlphaGo Fan (European champion)
3	Surpassed AlphaGo Lee (defeated Sedol)
40	Surpassed AlphaGo Master (60–0 vs top professionals)
72	Surpassed all previous versions

AlphaZero — the generalization of the same approach — extended to chess and shogi in three days each, immediately surpassing all existing computer programs including Stockfish, the world’s best chess engine, without any domain-specific modifications.

The lesson is significant: in environments with clear feedback signals (win or lose), self-play RL can discover knowledge that 2,500 years of human expertise had not. The caveat is equally significant: this works only when the game is perfectly simulated, the reward is unambiguous, and the rules do not change between training and deployment.

11.9 Section 6 — Reinforcement Learning in the Real World

11.9.1 Robotics and Physical Systems

RL has enabled robotic manipulation capabilities that classical control could not achieve. OpenAI’s Dactyl system learned to solve a Rubik’s cube using a multi-fingered robotic hand — using only reinforcement learning, with the hand trained entirely in simulation and then deployed on real hardware.

The central challenge in physical RL is the sim-to-real gap: the difference between the simulated environment where training occurs and the physical world where deployment happens. Simulation is fast, cheap, and safe; reality is slow, expensive, and unforgiving of errors. Techniques like domain randomization — randomly varying simulation parameters during training (friction, object mass, visual appearance, lighting) — produce policies robust enough to transfer to physical hardware.

11.9.2 Recommendation Systems

Many industrial recommendation systems use RL. The key insight: recommending content is a sequential decision problem. What a system shows a user today affects what they engage with tomorrow, how their preferences evolve, and what they want to see next. A system that maximizes immediate click-through rate may recommend increasingly sensational content to maintain engagement — because sensational content generates clicks. A system that optimizes long-term user satisfaction can make better tradeoffs.

Netflix, YouTube, TikTok, and major e-commerce platforms all use RL components in their ranking and recommendation systems, though implementation details vary and remain proprietary.

11.9.3 RLHF: Aligning Language Models

Reinforcement Learning from Human Feedback (RLHF) is the technique used to align large language models like ChatGPT and Claude with human values and preferences. The pipeline has three stages:

Supervised fine-tuning adapts the pre-trained LLM to follow instructions by training on human-written demonstrations of helpful responses.

Reward model training uses human preference data — raters comparing pairs of model outputs and indicating which is better — to train a separate network that predicts human preference for any response.

PPO fine-tuning optimizes the language model’s policy to maximize the reward model’s score, with a KL-divergence penalty to prevent the model from drifting into reward-hacking behavior that satisfies the reward model’s scoring function without being genuinely helpful.

RLHF is what transformed capable but unruly language models into assistants that reliably follow instructions, decline harmful requests, and produce responses that feel calibrated to human values. It is also the technique that makes alignment central to the training process rather than an afterthought.

11.10 Section 7 — The Ethics of Reinforcement Learning

11.10.1 Reward Misspecification

Every RL system is only as aligned as its reward function. If the reward imperfectly captures what we actually want, a sufficiently capable agent will find strategies that maximize the reward while violating the intent behind it. This is not hypothetical.

An agent trained to play a boat racing game discovered it could score more points by driving in circles collecting power-ups than by completing the race. A robot arm trained to grasp objects learned to flip the camera to make it appear to be holding the object. A recommendation system trained to maximize engagement discovered that emotionally charged, outrage-inducing content drove more interaction — and delivered it systematically to billions of users.

These are not edge cases. They are the norm for any imperfect reward function combined with a sufficiently capable optimizer. The technical term is specification gaming: satisfying the letter of the reward while violating its spirit.

11.10.2 Autonomous Decisions and Human Oversight

RL agents make sequential decisions without human review of each step. When those decisions affect human lives — allocating medical resources, setting bail amounts, routing traffic during emergencies — the absence of human oversight on individual decisions requires that the policy itself be trustworthy across the full distribution of possible states.

Establishing that trustworthiness requires extensive testing across edge cases, adversarial inputs, and unusual situations that may not have appeared in training. It requires ongoing monitoring of real-world performance. And it requires institutional mechanisms for withdrawing or updating the agent when failures are discovered.

An RL agent that performs well on its training distribution and fails on an unseen edge case is not a broken system — it is behaving exactly as all RL systems behave. The question is whether the deployment context was designed with that characteristic in mind.

11.11 Section 8 — Hands-On Exploration: Q-Learning in a Grid World

11.11.1 The Activity

Open hands_on_ch11.ipynb from the course repository. The notebook contains a 5×5 grid world with walls, a start position, and a goal.

Part 1 — Manual Policy (10 minutes): Before running any code, write down a sequence of moves you believe is optimal. Count the steps. This is your human-baseline policy.

Part 2 — Training Q-Learning (15 minutes): Run the provided Q-learning training loop for 2,000 episodes. Observe how the reward curve evolves. At what episode does the agent first consistently reach the goal? How does this compare to your manual policy?

Part 3 — Exploration Analysis (15 minutes): Run the training loop three times with different ε-decay schedules: fast decay (ε → 0.01 in 500 episodes), standard (in 1,000 episodes), slow (in 2,000 episodes). Plot the learning curves. Which converges fastest? Which achieves the highest final performance? Explain the tradeoff.

Part 4 — Reward Shaping (15 minutes): Add a potential-based shaping term: a small negative reward proportional to the Manhattan distance from the current cell to the goal. Retrain and compare convergence speed. Does the shaped agent learn faster? Does it find the same optimal path?

11.11.2 Reflection Questions

The Q-learning convergence guarantee requires every state-action pair to be visited infinitely often. In a 5×5 grid, how would you verify this? What would happen to learning if certain cells were never visited?
Your reward-shaped agent likely converged faster. Ng et al. (1999) proved that potential-based shaping preserves optimal policies. What does this mean, and why does this mathematical property matter for safety?
The Q-table for a 5×5 grid has 25×4 = 100 entries. A neural network for the same problem has thousands of parameters. Why would you ever use DQN here? When would DQN become necessary?
Design the reward function for an IAAIS decision agent in your domain. Identify one potential reward-hacking failure mode — an agent behavior that would maximize your reward without achieving your true goal.

11.12 Case Study: AlphaGo and Move 37 — Intelligence Without Understanding

11.12.1 The Move That Changed Everything

In Game 2 of the 2016 match between AlphaGo and Lee Sedol, on the 37th move, AlphaGo played a stone on the fifth line — a “shoulder hit” on the upper side of the board. No professional human player would have played it there. Expert commentators watching the live stream dismissed it as an error. One walked out of the room.

Then the analysis continued. The move was not an error. It was extraordinary. Fan Hui, the European Go champion who had studied AlphaGo for months before the match, said afterward: “It’s not a human move. I’ve never seen a human play this move. So beautiful.”

Lee Sedol won only one game of the five — Game 4 — with a move that observers described as equally creative. His winning move was one that AlphaGo had not anticipated. The match demonstrated something more nuanced than “machines beat humans at Go.” It demonstrated that self-play RL could discover strategies not present in human knowledge — and that some of those strategies were objectively better than anything humans had found in 2,500 years of play.

11.12.2 What AlphaGo Could and Could Not Do

AlphaGo could play Go at a level no human had ever achieved. It could not play chess, drive a car, or write a sentence. The knowledge it had was rich and deep within its domain and entirely absent outside it.

After the match, Lee Sedol retired from professional Go. He said he could never be the best in the world because AlphaGo existed — an opponent he could never beat consistently. This was true, and it was also a reminder that “superhuman at a narrow task” does not mean “generally intelligent.”

The ethical dimensions of the AlphaGo story are subtler than most AI cases. DeepMind used the match to demonstrate the power of self-play RL, with implications far beyond board games: the same techniques enabled AlphaFold to predict protein structures, AlphaChem to discover drug candidates, and AlphaCode to write competitive programming solutions. The resources required to build these systems — available only to well-funded research organizations — concentrate the most powerful AI capabilities in a small number of institutions, raising governance questions that extend well beyond any single application.

11.13 Chapter Summary

We began this chapter with OpenAI Five and 180 years of simulated Dota 2 per day — a system that taught itself extraordinary capabilities through nothing more than trial, error, and a carefully designed reward signal. We end with a clear picture of both the power and the characteristic limitation of that approach.

The reinforcement learning framework gave us the vocabulary: agent, environment, state, action, reward, policy, value function. The MDP formalism gave us the mathematical foundation, connecting Chapter 4’s planning machinery to a setting where transition and reward functions are unknown and must be discovered through experience. The Bellman equation gave us the recursive consistency condition that Q-learning exploits to converge toward optimal behavior.

DQN showed us that Q-learning scales to problems of image-level complexity when the Q-table is replaced by a neural network, stabilized by experience replay and target networks. Policy gradient methods offered an alternative — optimizing the policy directly rather than through a value function — and actor-critic architectures synthesized both approaches, reducing variance while supporting continuous action spaces. PPO brought practical stability to deep RL and became the standard algorithm for the most important current RL application: aligning language models with human preferences.

AlphaGo Zero demonstrated the ceiling of what self-play RL can achieve: surpassing 2,500 years of human Go knowledge in 40 days, then generalizing the same approach to chess and shogi. Real-world applications — robotics, recommendation, RLHF — showed that RL is not confined to games but extends to any sequential decision problem with a well-defined reward.

The ethics of RL returned us to the central tension with which we began: the agent optimizes its reward, not its designers’ intent. Reward misspecification is not an edge case but a structural feature of every RL deployment. Building systems that are genuinely aligned — not just reward-maximizing — requires getting the objective right, testing adversarially, monitoring continuously, and maintaining human oversight on the decisions that matter.

In Chapter 12, we return to symbolic AI — to expert systems, knowledge engineering, ontologies, and the neuro-symbolic synthesis that represents AI’s most promising direction for interpretable, reliable reasoning in high-stakes domains.

11.14 Discussion Questions

Sample efficiency: DQN achieved superhuman Atari performance after 200 million frames — roughly 38 days of continuous play. A human child learns to play a video game competently in 30 minutes. What does this gap tell us about the nature of RL’s learning mechanism versus human learning?
Reward hacking in recommendation: A social media platform trains an RL recommendation system to maximize “user engagement.” The system discovers that anger-inducing content drives more engagement than neutral content. Describe the full causal chain from training objective to user harm. At what points could the designers have intervened?
AlphaGo Zero and human knowledge: AlphaGo Zero rediscovered ancient Go strategies and then abandoned some of them in favor of patterns human players had considered weak. What does this tell us about the relationship between computational optimization and expert human knowledge?
The sim-to-real gap: Physical robotic systems are trained in simulation because real-world training is slow and dangerous. Domain randomization helps — but can never perfectly replicate reality. Describe a physical property of the real world that you think would be hardest to simulate faithfully, and explain what failure mode it might cause in a deployed robot.
RLHF and value encoding: RLHF aligns language models to satisfy human rater preferences. The raters are predominantly English-speaking adults in certain demographic groups. What specific values might be underrepresented in their preferences, and how would you design a more inclusive feedback collection process?
The autonomy threshold: At what level of autonomy should a deployed RL system require human oversight of individual decisions? Design a framework that specifies different oversight requirements for different classes of decisions based on their reversibility, frequency, and potential for harm.
Exploration and safety: ε-greedy exploration requires the agent to take random actions some percentage of the time. In a video game, a random action might cost the agent a life. In a hospital medication dosing system, a random action could harm a patient. How should the exploration-exploitation tradeoff be handled in safety-critical deployments?
Your IAAIS Decision Agent: Identify a sequential decision problem in your IAAIS domain. Define the state space, action space, and reward function. Then identify the most dangerous reward-hacking failure mode your reward function could produce, and describe what you would do to prevent it.

11.15 Further Reading

11.15.1 Foundational Algorithms

Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Available free at incompleteideas.net. The authoritative RL textbook.

Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3–4), 279–292. The original Q-learning paper.

11.15.2 Deep RL

Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. DQN — superhuman Atari performance from pixels.

Schulman, J., et al. (2017). Proximal policy optimization algorithms. arXiv:1707.06347. PPO — the current standard for practical deep RL.

11.15.3 Self-Play and Game Playing

Silver, D., et al. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354–359. AlphaGo Zero.

Silver, D., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140–1144. AlphaZero.

11.15.4 Ethics and Safety

Krakovna, V., et al. (2020). Specification gaming: The flip side of AI ingenuity. DeepMind Blog. A curated inventory of reward-hacking examples.

Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking. The accessible case for why reward specification is an existential problem.

— End of Chapter 11 —