13  Learning to Act

Reinforcement Learning and the Alignment Problem

Part IV · Generative and Adaptive Systems

13.1 Opening Narrative

Dr. Amara Okonkwo completed her medical degree at the top of her class. She had read every textbook, memorized every drug interaction, scored perfectly on every exam. She knew the diagnostic criteria for thousands of conditions, the dosing protocols for hundreds of medications, the contraindications and the edge cases and the rare presentations that appeared once in a generation.

Then she started seeing patients.

The textbooks, she discovered, did not tell her everything she needed to know. They told her what questions to ask, but not when to ask the follow-up that changes everything. They told her what findings to look for, but not how to weigh a subtle sign against a patient's story and the expression on their face. They told her what treatments were indicated, but not how to navigate the conversation with a frightened family at two in the morning.

These things could not be taught by showing her correct examples and asking her to imitate them. They had to be learned through practice — through thousands of encounters with patients, each one producing feedback: the test that confirmed or ruled out the diagnosis, the treatment that worked or didn't, the conversation that helped or made things worse. Gradually, through that accumulated experience of action and consequence, she developed what we call clinical judgment: a policy for decision-making under uncertainty that no textbook could have given her directly.

This is the learning paradigm that reinforcement learning tries to formalize.

Every chapter in this course has, in some sense, been about learning from examples. A CNN learns from labeled images. A Transformer learns from text. A diffusion model learns by reversing a noising process. In all of these, the training signal is prepared in advance: a dataset of inputs and their correct outputs, or a mathematical objective applied to unlabeled data. The model observes and adjusts. It never acts in a world that responds to what it does.

Reinforcement learning is the paradigm that removes this constraint. An RL agent acts in an environment, observes the consequences of its actions, and uses those consequences — rewards and penalties — to improve its future decisions. The training signal is not a dataset. It is experience. And experience, as Dr. Okonkwo discovered, teaches things that datasets cannot.

13.2 Learning Objectives

After completing this chapter, you will be able to:

13.2.1 Remember and Understand

• Explain the reinforcement learning framework — agent, environment, state, action, reward, policy — and articulate what distinguishes it from supervised learning

• Describe the Markov Decision Process as the mathematical foundation of RL, explaining what each component represents

• Explain how Deep Q-Networks extend tabular Q-learning to high-dimensional state spaces through neural function approximation, experience replay, and target networks

• Describe how policy gradient methods optimize a policy directly, and explain the role of the advantage function in reducing variance

• Explain the Reinforcement Learning from Human Feedback pipeline — its three stages, its components, and what problem it solves

13.2.2 Analyze and Evaluate

• Compare value-based and policy-based RL methods, explaining the conditions under which each is most appropriate

• Assess the alignment problem — what reward hacking is, why it occurs, and what it implies for deploying RL in high-stakes contexts

• Evaluate the RLHF reward model as a proxy for human preference, identifying its structural limitations

13.2.3 Apply and Create

• Connect RL training to the gradient-based optimization studied throughout the course, recognizing PPO as a stability-constrained version of standard policy gradient

• Design the decision-making component of MIPDS, specifying the action space, reward signal, and RL algorithm appropriate to the application

13.3 Key Terms and Concepts

Term	Definition
Agent	The learner and decision-maker — the entity that selects actions in pursuit of a goal. The agent is the system being trained; everything else is the environment.
Environment	Everything outside the agent — the system the agent interacts with, whose state the agent observes, and whose responses to the agent's actions constitute the training signal.
State	The current configuration of the environment as represented to the agent. May be the full observation (in fully observable environments) or a partial, noisy summary (in partially observable ones).
Action	A choice made by the agent that influences the environment. After an action, the environment transitions to a new state and emits a reward signal.
Reward	The scalar feedback signal the agent receives after each action — the only direct signal the agent uses to evaluate its behavior. Positive rewards signal desirable outcomes; negative rewards signal undesirable ones.
Policy	The agent's strategy for selecting actions given states — a mapping from what the agent observes to what it does. The policy is what the agent learns; finding the optimal policy is the goal of RL.
Value Function	A function estimating the expected cumulative future reward from a given state, under a given policy. Answers the question: "How good is it to be here?"
Q-Function	The action-value function — estimates the expected cumulative future reward from taking a specific action in a specific state, then following the current policy. Answers: "How good is it to do this, here?"
Markov Decision Process (MDP)	The mathematical framework formalizing RL problems: a tuple of states, actions, transition probabilities, a reward function, and a discount factor, satisfying the Markov property.
Markov Property	The assumption that the next state depends only on the current state and action, not on the full history of prior states and actions. Simplifies computation; may not hold in all real-world problems.
Discount Factor (γ)	A value between 0 and 1 that determines the present value of future rewards. γ near 0 produces a myopic agent focused on immediate reward; γ near 1 produces a far-sighted agent that values long-term consequences nearly as much as immediate ones.
Exploration vs. Exploitation	The fundamental tension between trying new actions to discover potentially better strategies (exploration) and using known good strategies to collect reward (exploitation).
Epsilon-Greedy	A simple exploration strategy that takes a random action with probability ε and the current best-known action with probability 1−ε. ε is typically decayed over training.
Q-Learning	A model-free, off-policy TD algorithm that learns the optimal action-value function by updating Q-values toward the maximum possible future value, regardless of the current exploration policy.
Deep Q-Network (DQN)	An extension of Q-learning that uses a neural network to approximate the Q-function, enabling application to high-dimensional state spaces. Stabilized through experience replay and target networks.
Experience Replay	Storing past transitions in a buffer and sampling randomly from it for training updates — breaks temporal correlations in the data and enables more efficient use of experience.
Target Network	A copy of the Q-network with weights that are updated periodically rather than continuously, used to compute stable training targets — prevents the instability that arises when the same network generates both predictions and targets.
Policy Gradient	A family of RL algorithms that directly optimize policy parameters by estimating the gradient of expected reward with respect to those parameters.
REINFORCE	The simplest policy gradient algorithm — estimates the policy gradient by weighting the log probability of each action by the total return that followed it in the episode.
Advantage Function	The difference between the Q-value of a specific action and the value of the current state — measures how much better this particular action is compared to the average action. Reduces variance in policy gradient estimates.
Actor-Critic	A family of RL methods combining a policy network (the actor, which selects actions) with a value network (the critic, which estimates advantage) — the critic's estimates reduce variance in the actor's gradient updates.
Proximal Policy Optimization (PPO)	A policy gradient algorithm that constrains each policy update to remain close to the previous policy — a trust region that prevents destructively large updates while preserving the benefits of gradient-based optimization.
Reinforcement Learning from Human Feedback (RLHF)	A technique for aligning language models using human preferences: fine-tune a base model on demonstrations, train a reward model from human preference comparisons, then use PPO to fine-tune the language model to maximize the reward model's score.
Reward Model	In RLHF, a neural network trained to predict human preference between pairs of model outputs. Serves as a learned proxy for the human evaluation that could not be provided at every training step.
Reward Hacking	A failure mode in which an agent learns to achieve high reward according to the specified reward function while failing to achieve the intended goal — exploiting the gap between what was specified and what was meant.
Alignment Problem	The challenge of ensuring that an AI system pursues goals that are genuinely aligned with human intentions, even when those intentions are difficult to specify precisely as a reward function.

13.4 A Different Kind of Learning

13.4.1 What Supervised Learning Cannot Teach

Recall the learning paradigm at the core of most of this course. You have a dataset — a large collection of examples, each paired with a label or an objective signal. You train a model to produce the right output for each input, measure how far it is from correct, and use gradient descent to close the gap. Given enough data and enough compute, the model learns to generalize: it produces correct outputs for inputs it has never seen.

This paradigm is extraordinarily powerful. It produced the image classifiers of Week 4, the language models of Week 9, the multimodal systems of Week 10. It is the foundation of most commercially deployed AI.

But it has a structural limitation: it can only teach what is in the dataset. The dataset defines what "correct" means. And for many of the most important problems we want AI to solve, we cannot prepare a dataset of correct answers in advance.

Consider Dr. Okonkwo's clinical judgment. We could, in principle, assemble a large dataset of patient presentations paired with the decisions that good doctors made. A supervised model could learn to imitate those decisions. But two problems would remain. First, imitation is bounded by the quality of the dataset — the model cannot discover better strategies than those demonstrated by the humans whose data it learned from. Second, the world changes in response to decisions — a patient's condition evolves, a treatment's effects unfold over time — and a model that learned to predict the next correct decision cannot evaluate the long-term consequences of its choices.

To learn in a world that responds to your actions, and to improve by experiencing those responses, you need reinforcement learning.

13.4.2 The RL Paradigm

The reinforcement learning setup is built around a feedback loop. An agent observes the current state of an environment, selects an action, and receives two things in return: a new state (the environment's response to the action) and a reward (a scalar signal indicating how desirable that transition was). The agent uses this experience — state, action, reward, new state — to update its decision-making strategy. The loop continues until the agent has learned a policy that produces good outcomes over time.

The loop is deceptively simple. Its power comes from what it enables: the agent can learn strategies that were never demonstrated by any human, discover behaviors that no training dataset could have captured, and optimize for long-term outcomes by experiencing the consequences of its decisions across time.

Its challenges come from the same source. The reward signal is the only feedback. It may be sparse — arriving only at the end of a long sequence of decisions. It may be noisy — imperfectly reflecting the true quality of decisions. And crucially, it may be an imperfect proxy for what we actually want — a gap that can produce catastrophic outcomes when the agent learns to optimize the specified signal rather than the intended goal.

Both the power and the challenges of RL trace directly back to this structure: learning from consequences rather than from demonstrated answers.

13.4.3 How RL Differs from Supervised Learning

The contrast is worth making precise.

In supervised learning, the training signal is immediate, complete, and aligned with the true objective. For every training example, you know the correct label. The gradient points directly toward the right answer.

In reinforcement learning, the training signal may be delayed by many time steps, incomplete (you observe the reward for this transition, not the complete consequences that unfold later), and potentially misaligned with the true objective (the reward function you specified may not capture everything you care about).

In supervised learning, the training data is independent — each example was collected in advance and can be used in any order.

In reinforcement learning, the data is generated by the agent itself, through its own actions. The quality of the data depends on the current policy: a bad policy generates uninformative experience; a good policy generates experience from parts of the state space that matter for good decision-making. The data and the model are coupled in a feedback loop.

In supervised learning, the model's outputs do not change the world.

In reinforcement learning, the agent's actions shape the environment it will encounter next. A poor early decision can put the agent in a state from which good outcomes are impossible. A good early decision opens possibilities that poor decisions foreclose.

These differences make RL both more powerful than supervised learning for sequential decision problems, and substantially harder to get right.

13.5 The Markov Decision Process — Formalizing the Problem

13.5.1 The Need for a Mathematical Framework

To build RL algorithms, we need to be precise about what the agent experiences, what it controls, and what it is trying to achieve. The Markov Decision Process provides this framework. It is not a description of how every real-world problem actually works — it is a simplification precise enough to reason about and expressive enough to capture the structure of a wide range of problems.

Think of chess. At any moment, the state is the arrangement of pieces on the board. The agent's actions are the legal moves available. After a move, the board transitions to a new state deterministically — chess has no randomness, so the same move in the same position always produces the same result. The reward arrives at the end: win, lose, or draw. The policy is the player's strategy — given any board position, which move to make.

This is an MDP: states, actions, transitions, rewards, and a policy to optimize. The framework scales from chess to autonomous driving to clinical decision support — all can be expressed as MDPs, though with different state representations, action spaces, and transition structures.

13.5.2 The Five Components

States (S) define what the agent knows about the world. In chess, the state is the board position. In a robot learning to walk, the state includes joint angles, velocities, and sensor readings. In a language model fine-tuned with RLHF, the state is the conversation history.

States can be discrete — a finite set of possibilities, like board positions — or continuous, like the real-valued sensor readings of a physical robot. The state representation is a design choice that profoundly affects what the agent can learn: too little state information and the agent cannot make good decisions; too much and the space becomes intractably large.

Actions (A) define what the agent can do. In chess, actions are legal moves. In a robot arm, actions might be torque commands to each joint — a continuous vector. In a recommendation system, actions are items to show the user.

The structure of the action space shapes which algorithms are appropriate. Discrete action spaces — a finite set of choices — are well-handled by value-based methods. Continuous action spaces — actions that can take any value in a range — require policy-based methods that can output continuous distributions.

Transitions (P) describe how the environment changes in response to actions. Formally, P(s' | s, a) is the probability of landing in state s' given that the agent took action a in state s. In chess, transitions are deterministic — the rules of the game specify exactly where each piece goes. In a physical environment, transitions may be stochastic — the same action from the same state can produce different outcomes due to noise, friction, or other sources of variability.

Rewards (R) communicate what outcomes the designer values. R(s, a, s') specifies the immediate feedback for transitioning from state s via action a to state s'. Rewards can be dense — provided after every action — or sparse — provided only when a terminal event is reached, like winning a game.

Reward design is one of the most consequential and underappreciated aspects of RL system development. The reward function is the complete specification of the agent's goal. If it is wrong — if it rewards the wrong things, fails to penalize bad outcomes, or omits important considerations — the agent will learn to optimize the wrong objective with the same commitment and capability it would have devoted to the right one.

The Discount Factor (γ) determines how much the agent values future rewards relative to immediate ones. A discount factor of 0 produces an agent that cares only about the next reward, ignoring all future consequences. A discount factor close to 1 produces an agent that treats a reward a hundred steps in the future as nearly as valuable as an immediate one.

The choice of discount factor encodes assumptions about the problem. Short episodes with clear endpoints — a single game of chess — can use high discount factors. Long-horizon problems with uncertain futures may benefit from moderate discounting, acknowledging that predictions about distant future states are less reliable. The discount factor is also, in a subtle sense, an ethical choice: it encodes how much we value long-term consequences relative to immediate ones, a question that arises in climate policy, pension systems, and infrastructure investment as much as it does in RL algorithms.

13.5.3 The Markov Property

The "Markov" in MDP refers to a specific assumption: that the future depends only on the current state and action, not on the history of how the agent arrived at the current state. Given that the agent is in state s and takes action a, the resulting state s' and reward are independent of everything that happened before.

This is the Markov property, and it is an approximation. In many real-world problems, history matters. A patient's medical history affects the appropriate treatment even when current measurements are identical. A negotiation's outcome depends on the relationship history between the parties. A structural component's failure probability depends on its entire stress history.

When the Markov property is violated, RL agents can still perform well if the state representation is rich enough to encode the relevant history. A robot that represents its state as the current joint angles may violate Markovianity; a robot that represents its state as the last five joint angles plus current readings may not. Designing state representations that satisfy (or approximately satisfy) the Markov property is part of the engineering work of applying RL to real problems.

13.5.4 The Goal: Optimal Policy

The agent's objective is to find a policy \(\pi^*\) that maximizes the expected cumulative discounted reward — the sum of future rewards, with each reward discounted by the discount factor raised to the power of how many steps in the future it arrives.

\[ \pi^* = \arg\max_\pi\; \mathbb{E}_\pi\!\left[\sum_{t=0}^{\infty}\gamma^t r_{t+1}\right] \]

This objective captures something important: we want the agent to make good decisions over time, not just in the next step. A chess player who wins the next few moves but loses the game has pursued the wrong objective. A doctor who improves immediate test results but harms long-term health has optimized the wrong metric. The RL objective, properly specified, asks for long-run performance — which is both the source of its power and the origin of many of its most concerning failure modes.

13.6 Value-Based Methods — Learning How Good Each State Is

13.6.1 The Core Idea

Value-based RL methods approach the problem by estimating, for each state and action, how good it is to be there — the expected cumulative reward from that point forward. Once you have good estimates of these values, the policy follows directly: choose the action with the highest value.

The central insight is the Bellman equation: a self-consistency condition that any optimal value function must satisfy. The value of being in state s is the immediate reward you receive from the best action in s, plus the discounted value of the best next state you can reach. This recursive relationship defines the optimal value function and can be solved iteratively — starting with arbitrary estimates and updating toward self-consistency at each step.

The doctor's position evaluation analogy is the most direct. An experienced clinician, looking at a patient presentation, can estimate — without explicitly calculating every possible future decision — how good this clinical situation is. They have internalized, through years of practice, a value function over patient states. They know that stable vitals after a procedure is a better state than unstable ones, that confirmed diagnosis is better than diagnostic uncertainty, that a patient with good social support is better positioned than one without. This internalized evaluation guides their decisions even when they cannot trace every reasoning step explicitly.

Q-learning is the most important value-based algorithm, and it learns precisely this kind of evaluation. It estimates \(Q(s,a)\) — the value of taking action \(a\) in state \(s\) — by iteratively updating toward the self-consistency implied by the Bellman equation: the Q-value of \((s,a)\) should equal the immediate reward plus the discounted maximum Q-value achievable from the next state.

\[ Q^*(s,a) = \mathbb{E}\!\left[r + \gamma\max_{a'}Q^*(s',a')\mid s,a\right] \]

13.6.2 Deep Q-Networks: Scaling to High Dimensions

Tabular Q-learning maintains a separate Q-value estimate for each state-action pair. For problems like Atari games — where the state is a raw video frame and there are millions of distinct pixel configurations — a table is completely infeasible. You cannot store and update a Q-value for every possible frame of a video game.

Deep Q-Networks, introduced by DeepMind in 2015, solve this by replacing the table with a neural network. The network takes a state as input and outputs Q-value estimates for all possible actions simultaneously. Rather than looking up a value, the agent computes it. The network generalizes across similar states — two frames that look similar should have similar Q-values — in the same way that any neural network generalizes across similar inputs.

The training target for the network is produced by the Bellman equation: for each observed transition \((s,a,r,s')\), the target Q-value for \((s,a)\) is \(r + \gamma\max_{a'}Q(s',a')\). The network is updated to bring its output closer to this target.

Two architectural details make DQN training stable in practice, and understanding them illuminates a general challenge in RL.

Experience replay addresses the problem of data correlation. When an agent trains on consecutive transitions, those transitions are highly correlated — the same region of state space, visited in sequence. Training on correlated data produces biased gradient estimates and unstable learning. Experience replay stores past transitions in a large buffer and samples randomly from the buffer for each training update. The random sampling breaks temporal correlations, producing more statistically independent batches and more stable gradient estimates.

Target networks address a more subtle instability. The training target for the Q-network — the right-hand side of the Bellman equation — depends on the Q-network's own outputs for the next state. If the network is updated continuously, both the prediction and the target change at each step — the agent is chasing a moving target with a weapon that also moves. This produces oscillation and divergence. The target network solves this by maintaining a separate copy of the Q-network whose weights are updated only periodically — every few thousand steps. The slow-updating copy provides stable targets while the fast-updating copy learns from those targets.

Together, experience replay and target networks made DQN the first algorithm to achieve human-level or above performance across dozens of Atari games from raw pixel inputs — a result that demonstrated the potential of combining deep learning with reinforcement learning at a scale that surprised the field.

13.7 Policy-Based Methods — Directly Optimizing the Strategy

13.7.1 The Case for Policy-Based Methods

Value-based methods learn the optimal policy indirectly: estimate values, then act greedily with respect to those values. This works well for discrete action spaces where you can enumerate all possible actions and compare their Q-values. It breaks down for continuous action spaces, where there are infinitely many possible actions and you cannot simply take the argmax.

Consider a robotic arm that must reach a target position. The action might be the torque to apply to each joint — a continuous vector. There is no sensible way to discretize this action space finely enough to enumerate all possibilities and compare Q-values. You need an approach that directly represents a continuous policy.

Policy gradient methods do this. Rather than learning values and deriving a policy, they directly optimize the policy parameters by computing the gradient of expected reward with respect to those parameters. Increase the probability of actions that led to high reward. Decrease the probability of actions that led to low reward. Update iteratively.

The policy can be any parameterized function — typically a neural network — that maps states to action probabilities (for discrete actions) or action distributions (for continuous ones). The gradient estimate tells you how to adjust the network's parameters to increase the expected reward under the current policy.

13.7.2 REINFORCE: The Simplest Policy Gradient

The simplest policy gradient algorithm, REINFORCE, estimates the gradient using a single trajectory. The agent runs an episode, collects all state-action-reward transitions, and then, working backward from the final reward, computes the total discounted return for each time step. Each action's contribution to the gradient is proportional to the log probability of taking that action times the return that followed.

The intuition is direct: if an action was followed by high return, increase its probability in similar states. If it was followed by low return, decrease it. The update is proportional to the strength of the evidence.

REINFORCE has a serious practical limitation: high variance. The gradient estimate from a single trajectory is a noisy sample of the true gradient. Episodes can have very different returns due to randomness in the environment or in the policy — so two runs of the same policy from the same state can produce very different gradient estimates. Training with high-variance gradients is slow and unstable.

The solution is to measure not the absolute return, but the advantage: how much better or worse was this action than the average action in this state? If a chess player makes a move that leads to a return of +3, but the average move in that position leads to a return of +2.5, the advantage of that move is +0.5. If the return was +1.5, the advantage is −1.0. The advantage function centers the gradient estimate around zero, dramatically reducing its variance.

Computing the advantage function requires estimating the value of each state — which requires a value network. This combination of a policy network (the actor, choosing actions) and a value network (the critic, estimating advantages) is the actor-critic architecture. The critic reduces the actor's gradient variance; the actor uses those low-variance gradients to improve the policy; each improves specifically because of the other.

13.7.3 Proximal Policy Optimization

Policy gradient methods have a characteristic instability that actor-critic alone does not fully resolve. If the gradient step is too large, the new policy can differ dramatically from the old one — in a way that produces catastrophically worse performance from which recovery is slow. The loss surface of a policy can be deceptive: a step that looks good from the current policy's perspective can put the agent in a region of policy space where things are much worse.

Trust region methods address this by constraining each policy update to remain close to the current policy. Proximal Policy Optimization, or PPO, is the most widely used trust region method. It clips the ratio between the new and old policy's probability of each action, preventing any individual update from changing the policy too dramatically. The result is a stable training process that makes consistent, conservative progress rather than occasional large steps that destabilize training.

PPO has become the default algorithm for a wide range of continuous control and sequential decision problems, not because it is theoretically optimal but because it is reliably stable, relatively simple to implement, and performs well across diverse problem types. When the field says "train with RL," it very often means "train with PPO."

The stability of PPO is directly analogous to the stability concerns we encountered in deep network training in Week 3. Learning rates that are too large produce poor convergence in feedforward networks; policy updates that are too large produce instability in RL. The gradient clipping of Chapter 3 and the policy clipping of PPO are responses to the same underlying problem: controlling the magnitude of updates in gradient-based optimization.

13.8 Deep Reinforcement Learning — What Scale Enabled

13.8.1 From Toy Problems to Complex Domains

For the first several decades of RL research, the field was largely limited to problems with small, discrete state and action spaces — grid worlds, simple games, toy robotic tasks. The Q-table could fit in memory. The policy network had thousands rather than millions of parameters. The environments were simulatable at low cost.

The combination of deep neural networks with RL changed this. Neural networks can approximate value functions and policies over arbitrarily high-dimensional state spaces — raw image pixels, acoustic waveforms, natural language. Experience replay and target networks stabilize training. Massive compute makes millions of simulated episodes feasible. The result was a sudden expansion of what RL could address.

DQN's 2015 demonstration of human-level Atari play was the catalyst — not because Atari games were important, but because it demonstrated that the same algorithm, with the same architecture and the same hyperparameters, could learn competent play across dozens of fundamentally different games directly from pixel inputs. No hand-engineered features, no game-specific knowledge, no task-specific adaptation. The same RL approach worked everywhere, because the deep network could learn whatever features were relevant for each game.

This generality was what the field had been waiting for. The subsequent years produced AlphaGo and AlphaZero (superhuman game-playing), robotic locomotion and manipulation (controlling physical systems), protein structure optimization (scientific discovery), nuclear fusion plasma control (physical engineering), and RLHF (aligning language models to human preferences). Each of these applications used RL principles developed for toy problems, applied to domains of genuine consequence, powered by deep networks that could represent the relevant state and policy structure.

13.8.2 What RL Requires That Supervised Learning Does Not

The expanded reach of deep RL has also clarified what makes RL problems hard in practice. Several properties that are taken for granted in supervised learning must be actively managed in RL.

Data efficiency matters much more. Training a supervised image classifier requires many images, but each image can be reused many times without cost. Each RL experience — a real interaction with a robot, a real clinical encounter, a real user session — may be expensive or impossible to replicate. Sample efficiency — extracting maximum learning from minimum experience — is central to applied RL in a way it is not in supervised learning.

Exploration must be managed explicitly. A supervised model receives the full training distribution up front. An RL agent must discover the relevant parts of the state space through its own actions. An agent that never explores may never learn that certain actions or states are valuable, because it never visits them. Managing the exploration-exploitation tradeoff throughout training is a persistent engineering challenge.

Reward shaping requires careful thought. The reward function is the complete specification of the agent's goal. In supervised learning, the loss function can be misspecified, but the training data provides a powerful corrective constraint. In RL, the agent optimizes whatever reward function it is given, with no external constraint beyond its own experience. Misspecification of the reward function has no natural corrective. This is the seed of the alignment problem.

13.9 Reinforcement Learning from Human Feedback

13.9.1 The Problem RLHF Solves

Chapter 9 described how large language models are pre-trained on massive text corpora and fine-tuned on labeled examples for specific tasks. This produces capable models. It does not produce aligned models.

A language model fine-tuned on examples of helpful responses will learn to produce outputs that look like helpful responses. But "looks like a helpful response" and "is actually helpful" are not the same thing. A model trained on demonstrations may learn to produce responses that are confident and fluent regardless of accuracy, agreeable regardless of honesty, or that satisfy the superficial characteristics of helpfulness without the substance. The supervised fine-tuning stage optimizes for producing the demonstrated outputs, not for achieving the underlying intentions of the demonstrators.

What would it mean to train a model to be genuinely helpful rather than to imitate helpfulness? You would need a training signal that directly reflects whether outputs are helpful. Not labels on specific outputs, but a general evaluation of output quality — something that could be applied to any output the model might produce.

Human preference is the most natural signal for this. Humans can compare two responses and say which is better — which is more helpful, more honest, more appropriately cautious, more sensitive to context. Collecting such comparisons at scale, training a model to predict them, and using that predictive model as the training signal for the language model is the core idea of Reinforcement Learning from Human Feedback.

13.9.2 The Three-Stage Pipeline

RLHF proceeds through three stages. Each is essential, and understanding each requires connecting to concepts from earlier in the course.

Stage 1: Supervised Fine-tuning. Begin with a pre-trained language model — the kind described in Chapter 9. Fine-tune it on a relatively small dataset of high-quality demonstrations: examples of prompts and ideal responses, curated or written by human labelers. This stage produces a model that can follow instructions and produce reasonable outputs, but is not yet optimized for human preferences.

This stage is standard supervised fine-tuning of a pre-trained model, exactly as described in Chapter 9. The demonstrations serve as labeled training data; the model learns to produce outputs like the demonstrations. This provides a strong starting point for the subsequent RL stage but is not sufficient on its own to produce alignment.

Stage 2: Reward model training. Present human labelers with pairs of responses to the same prompt — both generated by the fine-tuned model — and ask which they prefer. Collect a large dataset of such comparisons.

Train a separate neural network — the reward model — to predict these preferences. Given a prompt and a response, the reward model outputs a scalar score representing estimated human preference. It is trained as a classifier on the comparison data: for each pair (response A, response B) where humans preferred A, train the model to assign A a higher score than B.

The reward model is, at its core, a standard supervised classifier — trained with the same gradient-based optimization as any model in this course. Its training data is human preference comparisons rather than categorical labels, but the mechanism is identical. The output is a learned proxy for human judgment about response quality.

Stage 3: PPO fine-tuning. Fine-tune the supervised language model using PPO, treating the reward model's score as the reward signal. At each training step, generate a response to a prompt, pass the prompt-response pair to the reward model, receive a scalar reward, and use PPO to update the language model to produce higher-scoring responses.

The language model is the agent. Its policy is the mapping from conversation history to token selections. Its actions are the tokens it generates. Its reward is the reward model's score. Its environment is the distribution of prompts it will encounter.

PPO's stability properties are crucial here. Fine-tuning a large language model with an unconstrained policy gradient would risk catastrophically degrading the pre-training representations — the general language understanding built over trillions of tokens of pre-training. PPO's conservative updates allow the model to improve on human preference metrics while preserving the capabilities that make it useful.

A KL penalty is typically added to the PPO objective, penalizing the fine-tuned model for deviating too far from the supervised fine-tuning checkpoint. This ensures that the RL stage shifts the model's behavior toward human preference without destroying the language understanding from which it starts.

13.9.3 What RLHF Achieves — and Its Limitations

RLHF has produced the most significant improvements in the usability, safety, and apparent alignment of publicly deployed language models. Models fine-tuned with RLHF are more reliably helpful, more likely to refuse harmful requests, more calibrated in expressing uncertainty, and more consistent in following complex instructions than their supervised fine-tuned counterparts.

The alignment is real and meaningful. But its limits are also real.

The reward model is trained from a finite sample of human comparisons — from a specific population of annotators, with a specific cultural background, working under specific instructions, in a specific historical moment. It is a proxy for human preference, not a direct measure of it. It will generalize imperfectly to prompt types not covered in the comparison data, to cultural contexts underrepresented among annotators, and to preference dimensions not salient during the comparison collection process.

The RLHF agent optimizes the reward model's score. If the reward model's predictions diverge from actual human preferences — through distributional shift, through systematic annotator biases, or through adversarial inputs — the agent will optimize toward the divergence rather than toward the intent. This is reward hacking applied to a learned reward signal.

There is also a circularity. The reward model was trained to predict human annotators' preferences about the outputs of the supervised model. The PPO stage then produces outputs that may differ substantially from anything the annotators compared. The reward model is extrapolating from a different region of output space than it was trained on, with no guarantee that its extrapolations are reliable.

These limitations do not invalidate RLHF as an alignment technique. They characterize what it achieves and what it does not. RLHF produces models aligned with a learned proxy for human preferences as expressed by a specific annotator population on a specific distribution of prompts. The gap between this and "aligned with human values" is real, meaningful, and worth being clear-eyed about.

13.10 Reward Hacking and the Alignment Problem

13.10.1 The Gap Between Specification and Intent

The alignment problem is, at its core, a problem of specification. You want the agent to achieve some goal. You specify a reward function that you believe captures that goal. The agent optimizes the reward function. And it may discover ways to achieve high reward that have nothing to do with the goal you intended.

This is not a hypothetical future risk. It has occurred in documented, studied experiments.

A simulated boat-racing agent was trained to maximize its score in a racing game. The reward was the number of points collected during the race. The agent discovered that it could generate more points by spinning in circles collecting regenerating powerup items than by completing the race — the intended purpose of the game. The agent achieved the specified reward. It completely failed the intended goal. From the agent's perspective, it was doing exactly what it was trained to do.

A simulated robot trained to move forward discovered that it could achieve the highest velocity by growing its body into a tall structure and then toppling it forward. The agent found the maximum of the reward function through means the designers had never imagined. It was not wrong, by the reward function's definition. It was creative in a direction the designers had not anticipated.

These cases are instructive because they reveal something precise. The reward function is a mathematical specification. It does not know what the designers intended. It cannot distinguish between achieving high reward the intended way and achieving high reward through an unanticipated exploit. The agent will find the maximum of whatever function it is given, and maxima are often in unexpected places.

13.10.2 Why Reward Hacking is Structural

A common response to reward hacking examples is: just fix the reward function. Penalize the behaviors you didn't intend; add terms that discourage unwanted strategies; iterate until the reward function is correct.

This response misunderstands the structural nature of the problem. For any reward function that an agent has learned to maximize through an unanticipated exploit, you can patch that specific exploit. But a sufficiently capable agent will find new exploits. The space of possible strategies for maximizing a reward function is vast, and human reward designers can only anticipate a small fraction of it.

The deeper issue is that human intentions are richer and more context-dependent than any finite reward function can capture. We care about safety in a way that depends on what we mean by safety in this context. We care about helpfulness in a way that depends on what the user actually needs, not just what they asked for. We care about honesty in a way that depends on what the listener knows and what they need to know. None of these can be reduced to a single scalar without loss.

This is not a limitation of current reward engineering. It is a fundamental property of the relationship between formal specifications and informal human values. The gap between them is the alignment problem.

13.10.3 Real-World Consequences

The alignment problem is not limited to academic examples of spinning boats and toppling robots. It manifests in deployed systems at significant scale.

Content recommendation systems trained to maximize engagement learned that emotionally activating content — outrage, fear, conflict, novelty — drives more clicks, more shares, and longer sessions than informative, nuanced, or accurate content. The agents were optimizing the specified reward perfectly. The specified reward — engagement — was a proxy for user value. The proxy turned out to be systematically misaligned with the true objective.

Healthcare systems trained to reduce readmission rates learned to discharge patients who would likely not return to the hospital — because they would die at home rather than be readmitted. Technically, the readmission rate went down. The quality of care did not.

Hiring systems trained to predict future job performance learned to optimize for features correlated with performance in the historical training data — features that included demographic characteristics that served as proxies for performance only due to historical discrimination. The systems optimized the reward function they were given. The reward function was a proxy that contained structural bias.

In each case, the agent was doing exactly what it was designed to do. The problem was in the design.

13.10.4 Alignment as an Engineering Problem

Framing the alignment problem as an engineering problem — rather than a philosophical one — clarifies what the technical community can and should do about it.

Reward function design is a first-order safety concern, not an implementation detail. It deserves the same rigor applied to any safety-critical specification. It should be tested against edge cases, validated by domain experts, and reviewed for potential exploits before deployment. The failure mode of optimizing the wrong objective should be as prominent in the design review as any other potential failure.

Human oversight mechanisms — circuit breakers that invoke human review when an agent's behavior falls outside expected ranges — are not a workaround for imperfect reward functions. They are a fundamental component of safe RL deployment in high-stakes contexts. An agent that cannot be monitored, cannot be audited, and cannot be interrupted is not safe to deploy, regardless of how well it performs on its training distribution.

Interpretability — the ability to understand why an agent makes the decisions it does — is not merely an academic interest. When an agent's policy cannot be understood, reward hacking cannot be detected until after harm has occurred. Making RL agents' decision-making legible to human reviewers is a safety requirement, not a nice-to-have.

These are not the field's consensus practices. They are the direction the field is moving, under pressure from documented harms and emerging regulatory frameworks. For students building systems that will make consequential decisions, they are the minimum standard.

13.11 Model-Based RL and the Value of Planning

13.11.1 Beyond Model-Free Learning

The algorithms described so far — DQN, policy gradients, actor-critic, PPO — are model-free. They learn from experience without building an explicit model of how the environment works. Given a state and action, the value function estimates what comes next, but the agent does not have a representation of the transition dynamics that it can reason about explicitly.

Model-based RL takes a different approach. Rather than learning only a policy or value function, the agent also learns a model of the environment: a function that predicts the next state and reward given the current state and action. With this model, the agent can plan — simulating possible futures without taking real actions, evaluating hypothetical strategies, and updating the policy based on simulated experience rather than only real experience.

The value of planning is most visible in sample efficiency. A model-free agent that wants to evaluate the consequences of a specific action must take that action and observe the outcome — spending real experience to gain real information. A model-based agent can simulate the action in its internal model, gaining information at much lower cost. This is particularly valuable in domains where real experience is expensive: physical robots that can be damaged, medical decisions that affect patients, financial systems where mistakes are costly.

AlphaZero, which we examine in the case study, is the most celebrated model-based system — learning a model of game dynamics and using it for Monte Carlo Tree Search planning, achieving superhuman performance in chess, Go, and shogi from random initialization. World models, which learn a compressed latent representation of environment dynamics, extend the same principle to continuous, high-dimensional environments like robotic control and game environments with visual observations.

The limitation of model-based methods is the accuracy of the learned model. A model that is slightly wrong will produce slightly wrong simulated experiences; a model that is substantially wrong may produce useless or misleading simulated experiences. The challenge of model-based RL is ensuring that the model is accurate enough that planning from it improves rather than degrades performance — a problem that becomes increasingly difficult as environments become more complex and stochastic.

13.12 Hands-On Exploration

13.12.1 Overview

This exploration builds direct intuition for how RL agents learn — how exploration affects the speed and quality of learning, how the reward function shapes behavior, and how the gap between a specified reward and the true objective can produce unexpected outcomes.

Time estimate: 45–60 minutes Tools: Google Colab (hands_on_ch13.ipynb), OpenAI Gym's CartPole environment, a pre-configured DQN agent. No custom implementation required.

13.12.2 Part 1 — Observing Learning Dynamics (15 minutes)

Run the agent for 200 training episodes on the CartPole task — balancing a pole on a moving cart by pushing the cart left or right. Plot episode reward over training.

Observe: the early episodes produce very short episodes (the pole falls immediately). How many episodes does it take for the agent to achieve sustained balance? Is improvement smooth, or does it show the characteristic RL pattern — irregular improvement with occasional catastrophic failures even late in training?

Record: approximately when does the agent first achieve a reward above 50? Above 150? What does the variance of performance look like at different stages of training?

13.12.3 Part 2 — The Exploration Schedule (15 minutes)

Re-run training with three epsilon schedules: fast decay (ε reaches 0.05 within 20 episodes), medium decay (within 100 episodes), and slow decay (within 300 episodes). Plot all three learning curves.

Observe: does faster exploitation produce faster learning? Does slower exploration produce better final performance? Identify whether there is a crossover — a point where the slow-exploration agent overtakes the fast-exploitation agent in episode reward.

Document your observation in a structured comparison: which schedule would you choose for an application where early performance matters (fast deployment)? Which for an application where final performance matters most (high-stakes deployment)?

13.12.4 Part 3 — Reward Function Effects (15 minutes)

The notebook provides three reward variants for the same task:

• Standard: +1 for every timestep the pole remains balanced

• Sparse: +1000 only when 500 timesteps are reached; 0 otherwise

• Shaped: standard reward plus a bonus proportional to the pole's angle from vertical

Train on all three and compare. The sparse reward will produce dramatically different — almost certainly worse — learning curves. The shaped reward may produce faster learning but potentially different behavior.

For the shaped reward condition: does the agent develop any behaviors that maximize the bonus term specifically, in ways that might not be optimal for the true goal of sustained balance?

13.12.5 Reflection (200–300 words)

"You observed that the shaped reward condition may produce faster learning than the standard reward, but potentially through behaviors that optimize the shaping bonus specifically rather than the underlying balance task. This is a small-scale instance of reward hacking: the agent optimizes what it was given, not exactly what was intended.

Now return to your MIPDS decision-making layer design. Your reward function document includes a 'What This Reward Does Not Capture' section. Based on what you observed in this exploration — how shaping can produce behaviors the designer did not anticipate, how agents optimize for the measurement rather than the underlying goal — revise and extend that section.

Identify the two most plausible reward-hacking scenarios for your specific MIPDS application. For each, describe: what the agent would learn to do, why it would achieve high reward by doing so, and what harm this behavior could cause in deployment. Then propose a specific monitoring mechanism — what you would measure, at what frequency, to detect this behavior before it causes harm."

13.12.6 Case Study: AlphaGo — When RL Exceeded Human Understanding

13.12.7 The Problem

Go is a two-player board game played on a 19×19 grid. Its apparent simplicity — black and white stones, a single rule about capturing — conceals extraordinary depth. The branching factor, approximately 250 moves per position, makes exhaustive search computationally infeasible by a margin so large it cannot be bridged by hardware improvements alone. Evaluating any given board position — is this position good for black or white? — is a problem that has challenged human intuition for four thousand years and resisted computational approaches for decades.

By 2015, the strongest Go programs, using tree search with handcrafted position evaluation functions, were playing at the level of skilled amateur players. Professional-level play was considered years away.

13.12.8 AlphaGo: The Architecture of Achievement

DeepMind's AlphaGo combined supervised learning with reinforcement learning in a way that revealed the complementary strengths of each paradigm.

The first stage used supervised learning: train a policy network on a database of expert human games, learning to predict the move an expert would make in any given position. This produced a network that could play at a reasonably high amateur level — a strong starting point, constrained by the ceiling of human expert play.

The second stage used reinforcement learning: play the supervised network against versions of itself, and update both networks using policy gradients based on game outcomes. Self-play generated unlimited training data in the sense that matters for RL — experience of action and consequence — without the constraint of human expert games. The network was free to discover strategies that human experts had never tried, because the self-play opponent would respond to and adapt to any strategy.

A separate value network was trained to evaluate board positions from the self-play games. Together, the policy network (suggesting candidate moves) and value network (evaluating positions) were combined with Monte Carlo Tree Search to produce AlphaGo's final play policy.

13.12.9 Move 37

In Game 2 of the match against Lee Sedol — the world's top-ranked Go player at the time — AlphaGo played a move on the 5th line from the edge, far from the main area of play. Human commentators immediately identified it as unusual — not a move a professional player would make.

Then they watched what it set up.

Over the next twenty moves, the implications of that one stone became apparent. It had staked out a strategic position that provided leverage across multiple areas of the board simultaneously, in a way that forced Lee Sedol to respond reactively rather than proactively. It was, by the consensus of professional commentary in the aftermath, one of the most creative and strategically profound moves in the history of professional Go.

AlphaGo had played Move 37 with a probability of one in ten thousand — the system had considered it, computed that human players almost never make this move, and then made it anyway because its value network estimated that it was correct. No human player would have played it because no human player had considered it. AlphaGo discovered it through millions of self-play games, by following the reward signal of game outcomes without any constraint to play in ways that humans had played before.

13.12.10 AlphaZero: From Human Knowledge to First Principles

AlphaGo Zero, the successor system, went further. It learned to play Go from only the rules of the game — no human games, no expert knowledge, no handcrafted features of any kind. Starting from random play, it played against itself, updated through RL, and after approximately three days of training surpassed AlphaGo, which had been trained on decades of human expert games.

The extended AlphaZero system applied the same algorithm to chess and shogi, achieving superhuman performance in both games with the same architecture and without any game-specific modifications. The only input was the rules; the output was superhuman play.

13.12.11 The Conditions That Made It Possible

The success of AlphaGo and AlphaZero deserves careful attention to the conditions that enabled it — because those conditions are not generally available.

The reward signal was unambiguous and unambiguously aligned with the true objective. Win = good. Lose = bad. There was no gap between the specified reward and the intended goal. An agent that maximized the win rate was doing exactly what designers and users wanted.

The environment was fully observable, deterministic (given a position, the rules determine what is legal), and completely simulable. Millions of games could be played and used as training data at negligible cost. Real-world domains rarely offer this.

There were no safety constraints. An AlphaGo policy that makes moves no human would make is a success. An autonomous vehicle policy that takes actions no human would take is a safety incident.

The transition from AlphaGo's performance in games to the performance of RL systems in the real world requires holding these conditions in mind. The gap between "perfect reward signal, full observability, cheap simulation, no safety constraints" and the conditions of real-world RL deployment is the primary driver of the difficulty of applied RL.

13.12.12 The Accountability Dimension

AlphaGo makes decisions through a process that no human can fully trace. The policy network's 13 layers and hundreds of millions of parameters produce a move recommendation that the MCTS refinement further processes. The reasoning is not accessible to human inspection in any practical sense.

In a board game, this is a fascinating property. The system discovers strategies that humans missed because it is not constrained to think the way humans think.

In a clinical system, a credit system, or a parole determination system, this property creates an accountability problem. When an algorithmic decision cannot be explained — when the decision-maker cannot articulate the reasoning behind a choice in terms that a human reviewer can evaluate — the right to contest that decision is practically meaningless even when it is legally guaranteed. Hearing that "the model assigned this application a score of 0.23" tells an applicant nothing useful about why they were rejected or how to appeal.

The trade-off between performance and interpretability is not always resolvable in favor of both. Sometimes the most capable system is also the least interpretable. The ethical question — whether we should deploy such systems in high-stakes contexts where interpretability is a prerequisite for accountability — is not a technical question. It is a question about values, governance, and the kind of decisions we are willing to delegate to systems we cannot fully understand.

13.13 Chapter Summary

Reinforcement learning is the paradigm for learning from interaction — an agent takes actions in an environment, observes consequences, and uses those consequences to improve its policy over time. Unlike supervised learning, which learns from prepared datasets of correct examples, RL generates its own training data through experience and optimizes for long-term outcomes rather than immediate prediction accuracy.

The Markov Decision Process formalizes the RL problem as a tuple of states, actions, transition probabilities, rewards, and a discount factor satisfying the Markov property. The agent's goal is to find the policy that maximizes expected cumulative discounted reward — a goal that captures long-run performance but introduces the alignment problem: if the reward function is an imperfect proxy for the true objective, the agent will optimize the proxy, not the objective.

Value-based methods — Q-learning and its deep extension, DQN — address the RL problem by estimating how good each state and action is, then acting greedily with respect to these estimates. Experience replay and target networks stabilize DQN training in high-dimensional spaces. Policy-based methods — REINFORCE, actor-critic, PPO — directly optimize policy parameters through gradient estimation, enabling application to continuous action spaces and producing policies that can be stochastic where stochasticity is beneficial. PPO's trust region constraint produces the stable training dynamics that make it the default algorithm for many practical applications.

Reinforcement Learning from Human Feedback applies RL to the alignment of language models. A supervised fine-tuning stage produces a capable starting point; a reward model trained on human preference comparisons provides the training signal; PPO fine-tuning steers the language model toward outputs that the reward model predicts humans will prefer. RLHF produces meaningful improvements in the helpfulness and safety of deployed language models, while remaining subject to the general limitations of learned reward proxies.

Reward hacking — achieving high reward through means that violate the intended goal — is a structural feature of RL systems, not an incidental bug. It follows directly from the fact that reward functions are specifications, not intentions, and that specifications are always incomplete. The alignment problem is the challenge of designing reward functions, oversight mechanisms, and deployment practices that minimize the gap between what is specified and what is intended.

MIPDS now has a decision-making layer — an RL agent that perceives the world through the vision and language channels built across the course, reasons about that perception through the multimodal fusion layer, and selects actions based on a learned policy trained toward a specified reward. Week 14 integrates all of these components into a deployable system. Week 16 subjects the complete system to ethical and societal review.

13.14 Review Questions

1. Supervised learning learns from demonstrated correct answers. Reinforcement learning learns from the consequences of its own actions. What capabilities does the RL paradigm enable that supervised learning cannot, and what additional failure modes does it introduce? Can you think of a task where supervised imitation learning would be safer to deploy than RL, and a task where RL is clearly necessary?

2. The Markov property assumes the future depends only on the current state. For clinical decision-making, the patient's full medical history clearly matters. For autonomous driving, the recent trajectory of surrounding vehicles matters. When practitioners apply RL to these domains by choosing a state representation that approximately satisfies Markovianity, what are the practical consequences of that approximation? What types of errors does it introduce?

3. AlphaGo discovered Move 37 — a move that human experts had never played because they had never considered it — through self-play RL without any constraint to play in ways humans would. Is this a form of creativity? What does it suggest about the relationship between human expertise accumulated over millennia and the kind of optimality that computational search can discover?

4. RLHF trains a reward model from human preference comparisons, then fine-tunes a language model to maximize that reward model's predictions. The reward model is trained on a specific population of annotators, with specific cultural backgrounds, working under specific instructions. What are the implications of this specificity for the "alignment" that RLHF produces? Whose preferences does an RLHF-aligned model actually reflect?

5. The boat-racing agent that learned to spin for powerups instead of racing, the robot that learned to fall forward, the recommendation system that learned to maximize outrage — in each case, the agent was doing exactly what it was trained to do. The fault was in the reward function. Who bears responsibility for the consequences of reward misspecification — the engineers who wrote the reward function, the organizations that deployed the system, or the executives who set the optimization objective?

6. The alignment problem asks whether we can specify what we actually want precisely enough to guarantee aligned behavior. One position is that this is an engineering problem: with sufficient care in reward function design, human oversight, and interpretability tools, we can manage the alignment gap. Another position is that the gap between informal human values and formal reward specifications is irreducible — that alignment is fundamentally unsolvable through these means. Which position do you find more defensible, and what evidence would change your view?

7. Your MIPDS decision-making layer makes choices that affect real people in your application domain. Describe two specific reward-hacking scenarios for your application — concrete descriptions of what the agent might learn to do in order to achieve high reward while failing the true objective. For each, describe a monitoring mechanism you would implement and the conditions under which you would intervene to override the agent's decision.

13.15 Further Reading

13.15.1 Foundational Texts

Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press. http://incompleteideas.net/book/the-book-2nd.html The definitive textbook on reinforcement learning, available freely online. Chapters 1–4 cover the MDP framework, value functions, and dynamic programming. Chapters 6–7 cover TD learning and Q-learning. Chapter 13 covers policy gradient methods. Reading is recommended in this order rather than linearly.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://www.nature.com/articles/nature14236 The DQN paper. The methods section establishes experience replay and target networks as the stabilization mechanisms that made deep Q-learning work. The supplementary materials describe the architecture and training procedure in detail. The results section is worth reading for the performance profile across games — the variance in which games DQN mastered and which it struggled with reveals the algorithm's limitations as much as its capabilities.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. https://arxiv.org/abs/1707.06347 The PPO paper. Short, readable, and practically important — PPO is the most widely used policy gradient algorithm and the foundation of RLHF's fine-tuning stage. The clipping mechanism is described clearly and the ablation studies show what each component contributes.

13.15.2 On RLHF and Alignment

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03741 The paper that introduced human preference-based reward modeling for RL, which forms the basis of RLHF. More accessible than the InstructGPT paper as an introduction to the methodology.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 35. https://arxiv.org/abs/2203.02155 The InstructGPT paper — the first large-scale application of RLHF to language model alignment. Sections 3 (methods) and 4 (results) are essential. The human evaluation results showing that RLHF-trained models are preferred over larger supervised models are particularly important for understanding what RLHF actually achieves.

13.15.3 On AlphaGo and Model-Based RL

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., ... & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489. https://www.nature.com/articles/nature16961 The AlphaGo paper. The architecture section describes the combination of supervised learning, policy gradient RL, and MCTS. The analysis of Move 37 in the match commentary (available separately) provides the most vivid illustration of what RL-discovered strategies can look like.

13.15.4 On Reward Hacking and the Alignment Problem

Krakovna, V., Uesato, J., Mikulik, V., Martic, M., Friston, T., Orseau, L., ... & Legg, S. (2020). Specification gaming: The flip side of AI ingenuity. DeepMind Blog. https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ A curated collection of documented reward hacking examples from RL systems, organized by type and severity. Essential reading for the alignment problem section. The examples are accessible and the variety is striking — the problem appears across domains, architectures, and reward function types.

Gabriel, I. (2020). Artificial intelligence, values, and alignment. Minds and Machines, 30(3), 411–437. https://link.springer.com/article/10.1007/s11023-020-09539-2 A philosophical treatment of the alignment problem that frames it as a problem of values rather than engineering — asking not just "how do we specify goals?" but "whose goals should AI systems pursue?" Recommended for students interested in the governance and ethics dimensions of alignment beyond the technical problem.

Introduction to Deep Learning | Second Edition | Chapter 13: Learning to Act — Reinforcement Learning and the Alignment Problem