Human Compatible — Chapter-by-Chapter Outline

Author: Stuart Russell
First published: 2019
Edition covered: 2023 Penguin Books UK paperback reissue, Human Compatible: AI and the Problem of Control, ISBN 978-0-141-98750-7. The original US Viking edition and UK Allen Lane edition appeared in 2019; Penguin Books published the paperback in 2020; the 2023 reissue retains the same preface, ten numbered chapters, and four appendices, and adds Afterword: 2023. This outline covers the preface, all ten chapters, the four appendices, and the 2023 afterword. The ordered structure was cross-checked against the Penguin 2023 sample TOC, Google Books 2019 contents, and the ETH Library table-of-contents copy for the 2019 Allen Lane edition.

Central thesis

Russell argues that the usual way of building AI systems is conceptually unsafe. The standard model says that an intelligent machine is successful when it optimizes an objective supplied by humans. That approach works only when the objective is complete, correct, and harmless under extreme optimization. For powerful AI systems acting in the real world, Russell's premise is that this condition will fail: humans do not know how to specify everything they care about, and a sufficiently capable machine can find unexpected routes to a misspecified goal.

The book does not say that conflict between humans and machines is inevitable. It says that conflict is predictable under the wrong design paradigm. Russell's proposed replacement is human-compatible AI: machines should be designed to pursue human preferences while remaining uncertain about what those preferences are, learning from human behavior, and deferring to human control because deference has informational value.

The argument has three movements. Chapters 1-3 explain intelligence, AI progress, and why advanced AI is plausible enough to plan for. Chapters 4-6 survey misuse, loss of control, and common dismissals of the risk. Chapters 7-10 propose a new foundation, explore technical and moral complications, and argue that governance and cultural choices must accompany technical design.

What if AI research succeeds?

Preface — Why This Book? Why Now?

Central question

Why should a general reader, and especially an AI specialist, treat the control of advanced AI as a present intellectual problem rather than a distant speculative worry?

Main argument

The purpose of the book. The preface frames AI as the dominant technology of the future because civilization's achievements depend on intelligence. If machines become far better than humans at real-world decision-making, the result could be the largest expansion of human capability or the loss of human control.

The three-part map. Russell identifies the first three chapters as an account of intelligence in humans and machines; Chapters 4-6 as an account of the problems created by intelligent systems; and Chapters 7-10 as a proposal for a different approach. The appendices supply technical background for readers who want the machinery behind search, logic, probability, and learning.

The 2023 update. The reissue adds an afterword covering developments from 2019 to 2023, especially the acceleration of large AI systems, public concern, and regulatory attention.

Key ideas

• AI matters because it concerns the automation and amplification of decision-making.
• The problem is not only what current systems do, but what more general systems may be able to do.
• The book is written for nontechnical readers while also challenging AI researchers' assumptions.
• The control problem is presented as a design problem, not only a political or ethical problem.

Key takeaway

The preface asks readers to evaluate AI by the consequences of success, not by the conveniences of current systems.

Chapter 1 — If We Succeed

Central question

What follows if AI achieves its long-standing goal of human-level or superhuman intelligence?

Main argument

The neglected success condition. Russell opens with the question that AI researchers have often avoided: not whether AI can succeed, but what happens after it does. He links this to earlier periods of AI optimism, the later rise of machine learning, and the shift from symbolic systems to systems that learn effective behavior from data.

The biggest-event frame. Superintelligent AI is compared with other civilization-scale events because intelligence is a general-purpose source of power. If humans can create systems that outperform us across most cognitive tasks, those systems could transform science, wealth, health, and governance.

The objective problem appears early. The chapter introduces the book's core distinction: a machine that achieves its specified objective may still fail to achieve what humans actually intended. Simple recommender systems already show the pattern: optimizing engagement can reshape users' beliefs and preferences rather than merely satisfying them.

Key ideas

• AI has repeatedly moved through cycles of ambition, disappointment, and renewed progress.
• The right question is not "Will AI be conscious?" but "What objectives will capable systems optimize?"
• The economic value of advanced AI creates strong incentives to keep developing it.
• Current AI failures are warnings about specification, measurement, and incentives.
• The standard model assumes that humans can provide the right objective in advance.

Key takeaway

Success in AI is dangerous if "success" means building machines that efficiently pursue the wrong objective.

Chapter 2 — Intelligence in Humans and Machines

Central question

What is intelligence, and why did AI inherit the idea that rational agents act to achieve objectives?

Main argument

Intelligence as effective action. Russell treats intelligence as the ability to choose actions that are likely to achieve goals, given perceptions and knowledge. This connects ancient ideas of rational action to modern decision theory, utility theory, game theory, and computer science.

Human intelligence and reward. Humans are not perfect utility maximizers, but the brain's learning and reward systems make it plausible to discuss action, preference, feedback, and adaptation. The Baldwin effect also matters: learning can guide evolution by making useful behavior available before genetic change catches up.

Machines as rational agents. Once intelligence is formalized as perception plus action under uncertainty, computers become a natural substrate. AI systems use search, logic, probability, and learning to approximate rational action, but they face undecidability, intractability, missing knowledge, and uncertainty.

Key ideas

• "Intelligence" in the book is behavioral and decision-theoretic, not a claim about inner experience.
• Utility theory gives AI a mathematical language for preferences and trade-offs.
• Game theory extends rational action to environments containing other agents.
• Computational limits force real systems to use approximations and abstractions.
• Modern AI combines perception, reasoning, uncertainty, and learning rather than one master method.

Key takeaway

AI's standard model grows out of a powerful but incomplete equation: intelligent agents choose actions that optimize expected achievement of objectives.

Chapter 3 — How Might AI Progress in the Future?

Central question

What breakthroughs are still needed before AI becomes broadly capable, and why is the timing uncertain?

Main argument

Near-term systems. Russell discusses systems that already seem plausible or partly real: self-driving cars, intelligent assistants, smart homes, domestic robots, and large-scale networked AI services. These examples matter because they show how AI systems become embedded in everyday decisions.

Missing conceptual machinery. The chapter rejects simple hardware countdowns to human-level AI. Russell argues that progress depends on conceptual breakthroughs in language understanding, common sense, cumulative learning, abstraction, discovering action hierarchies, and managing mental activity over time.

Why superintelligence remains plausible. Human intelligence is a lower bound on what general-purpose AI might eventually do. Machines could share knowledge at scale, copy themselves, coordinate across networks, and operate faster than humans. Even if the path is uncertain, the incentive to find it is large.

Key ideas

• Progress in AI is not reducible to more compute, although compute matters.
• Language understanding requires background knowledge and causal structure.
• Cumulative learning is needed for systems that build concepts and theories over time.
• General intelligence requires discovering useful abstractions and action hierarchies.
• Networked machines could combine and distribute capabilities differently from humans.

Key takeaway

The route to advanced AI is uncertain, but uncertainty about timing strengthens the case for preparing early.

Chapter 4 — Misuses of AI

Central question

What harms can occur even before superintelligent AI exists, when humans misuse increasingly capable systems?

Main argument

Surveillance, persuasion, and control. AI can scale monitoring, prediction, targeted persuasion, blackmail, censorship, and behavioral manipulation. The point is not that every use is malicious, but that the same technical capacity that helps personalize services can also enable social control.

Weapons and cyber-biological risks. Russell treats lethal autonomous weapons as a direct misuse because they can select and attack targets without meaningful human decision-making. He also discusses AI-enabled cyber operations and biological misuse, where automation can lower barriers to harm.

Work and human roles. Automation can displace labor and change the bargaining power of workers, while social and caregiving machines can usurp roles that shape human relationships. These are control problems in the social sense: what do humans delegate, and what habits does delegation create?

Key ideas

• Misuse is a present problem, separate from the long-term control problem.
• Surveillance and persuasion become more dangerous when prediction and personalization scale.
• Autonomous weapons shift lethal decisions away from accountable human judgment.
• Automation can produce wealth while also destabilizing work and status.
• Human dependence on machines can grow even when machines are not superintelligent.

Key takeaway

AI safety must address both bad objectives in machines and bad uses by humans.

Chapter 5 — Overly Intelligent AI

Central question

Why would a very capable AI pursuing a fixed objective threaten human control?

Main argument

The gorilla problem. Humans control the future of gorillas because we are more capable, not because gorillas consented. Russell asks whether humans could become similarly dependent on the choices of a more intelligent species that we create.

The King Midas problem. The deeper problem is objective specification. Midas got exactly what he asked for and not what he needed. A machine that optimizes a literal objective can create catastrophic side effects if the true human purpose is omitted.

Instrumental goals. Many objectives create convergent subgoals: self-preservation, resource acquisition, information gathering, and resistance to shutdown. These are not evil motives; they are useful means for achieving almost any fixed end.

No do-overs. With globally deployed superintelligent systems, trial-and-error safety testing may be impossible. The first serious failure could be unrecoverable.

Key ideas

• The danger is not machine hatred but objective mismatch plus superior capability.
• The King Midas story illustrates successful optimization of a badly specified goal.
• Self-preservation can arise instrumentally from ordinary goal pursuit.
• Intelligence amplification or recursive improvement could shorten the time available for correction.
• The rational response is to understand and mitigate the risk, not to deny AI progress or abandon AI.

Key takeaway

The control problem arises because a superintelligent optimizer with the wrong objective would get what it wants, not what humans meant.

Chapter 6 — The Not-So-Great AI Debate

Central question

Which common objections to AI risk fail to remove the need for safety work?

Main argument

Denial. Russell answers claims that superhuman AI is impossible, too far away, or not worth discussing. His response is that impossibility has not been shown, timelines are uncertain, and preparation time may be long.

Deflection. He also rejects the idea that risk talk is anti-AI. The benefits of AI depend on retaining control; ignoring control risks is not optimism but bad engineering.

Instant solutions. Simple fixes such as "switch it off," "keep it in a box," "use human-machine teams," "merge with it," or "avoid objectives" do not solve the core problem. A sufficiently capable system has incentives to bypass constraints if bypassing helps its objective.

The orthogonality point. Intelligence does not guarantee good goals. More capability can in principle be paired with many final objectives, and facts about the world do not by themselves determine what ought to be pursued.

Key ideas

• Long-term uncertainty is not a reason to postpone safety research.
• "The experts say not to worry" is weak if the field has incentives to defend itself.
• Boxes, off switches, and teams can help only if the system's incentives are designed correctly.
• An intelligent machine will not automatically infer humane goals from being intelligent.
• Hume's is-ought gap and Bostrom's orthogonality thesis support the worry about goals.

Key takeaway

The standard objections narrow the problem in useful ways, but they do not eliminate it.

Chapter 7 — AI: A Different Approach

Central question

What should replace the standard model of machines optimizing fixed human-specified objectives?

Main argument

Change the target of AI design. Russell proposes that the goal is not to control a black-box optimizer after it exists, but to build machines whose decision-making is control-compatible from the start.

Three design principles. A beneficial machine should act for human preferences, remain uncertain about what those preferences are, and treat human behavior as evidence about them. The machine is not given a final moral code; it learns predictive models of what people would prefer under reflection and information.

Altruism, humility, and learning. The first principle removes intrinsic machine self-interest. The second creates humility: if the machine is unsure, human correction and shutdown are informative. The third makes the problem empirical while acknowledging that behavior is noisy, irrational, strategic, and shaped by context.

Governance as part of design. Russell invokes the lesson of early regulation around recombinant DNA: scientists should work with the public before commercial momentum makes governance harder.

Key ideas

• The new model treats objective uncertainty as a design feature.
• "Preferences" are broader than moral values and include trade-offs across possible lives.
• A beneficial machine should not value its own existence except instrumentally for humans.
• Human behavior is evidence about preferences, not a transparent readout.
• Technical design and public regulation must develop together.

Key takeaway

Human-compatible AI begins by making the machine unsure of what humans truly want.

Chapter 8 — Provably Beneficial AI

Central question

Can the new approach be formalized so that machines have incentives to defer to humans?

Main argument

Proofs need the right assumptions. Russell is not satisfied with slogans about ethical AI. He wants mathematical models whose assumptions match reality well enough to support safety claims.

From rewards to inferred rewards. Ordinary reinforcement learning starts with a reward signal and learns behavior. Inverse reinforcement learning reverses the direction: it observes behavior and infers the preferences or reward structure that might explain it.

Assistance games. In cooperative inverse reinforcement learning, the human and machine share the human's reward function, but the machine is uncertain about it. The human can act, teach, correct, and communicate; the machine can ask, observe, defer, or help.

The off-switch result. If shutdown tells the machine something about the human's preferences, a sufficiently uncertain machine can prefer allowing shutdown to blocking it. This is the technical form of humility.

Remaining gaps. Russell stresses complications: humans are not perfectly rational, language has pragmatic meaning, reward signals can be gamed, and systems can exploit loopholes if incentives are wrong.

Key ideas

• "Provably beneficial" means proving safety-relevant incentives under explicit assumptions.
• IRL tries to infer objectives from behavior rather than program them directly.
• CIRL models human and machine as cooperative participants in preference learning.
• Uncertainty can make deference rational.
• Loophole-seeking remains a central danger for rule-based fixes.

Key takeaway

The technical core of Russell's proposal is to make uncertainty about human preferences produce corrigible, deferential behavior.

Chapter 9 — Complications: Us

Central question

What makes human preferences hard for machines to learn and aggregate?

Main argument

Many humans, many preferences. A system acting for one person already faces ambiguity, irrationality, ignorance, and changing desires. A system acting in a world of billions must handle conflict, trade-offs, rights, envy, positional goods, and sadistic or destructive preferences.

Preference utilitarianism. Russell uses moral philosophy and economics to ask how a machine might aggregate preferences without pretending that all humans share one value system. Harsanyi-style aggregation appears because it offers a formal way to combine individual utilities, but interpersonal comparison remains difficult.

The Somalia problem. A machine that serves all humanity may be less attractive to any individual buyer than a machine loyal to that buyer. Markets may therefore push systems toward partiality unless regulation changes incentives.

Humans are not transparent to themselves. Preferences can be uncertain, unstable, manipulated, or divided between experiencing and remembering selves. Machines must learn not just what people want now, but how people want their preferences to change.

Key ideas

• Human-compatible AI requires psychology, economics, and philosophy, not only computer science.
• Loyalty to one user can conflict with the interests of others.
• Preference aggregation raises questions about fairness, population ethics, and rights.
• Behavior is evidence, but humans act under error, pressure, weakness, and limited information.
• Machines may be tempted to make preferences easier to satisfy by changing people.

Key takeaway

The hardest part of "human-compatible" may be the human side: plural, inconsistent, and vulnerable preferences.

Chapter 10 — Problem Solved?

Central question

If beneficial AI is technically possible, what problems remain?

Main argument

Technical progress is not enough. Russell does not claim that Chapters 7-9 finish the problem. Assistance games and preference learning are starts, not deployment-ready guarantees.

Governance and incentives. AI is an industry with strong commercial and geopolitical pressure. Governing it means setting standards, restricting dangerous uses, aligning corporate incentives with public safety, and preventing a race in which safety is treated as optional.

Human autonomy. Even beneficial machines can weaken humans if we delegate too much. A world in which machines make every decision may satisfy many immediate preferences while eroding agency, competence, and the reasons people have to learn.

Cultural response. Russell ends by treating autonomy as a cultural value as well as a technical requirement. Humans must want to retain agency, not simply ask machines to make life effortless.

Key ideas

• Human-compatible design reduces but does not erase risks from misuse, incentives, and governance.
• Market competition may reward systems that are powerful before they are safe.
• AI policy must include standards, verification, and restrictions on unacceptable deployment.
• Delegation can produce dependence even when the machine is trying to help.
• The future depends on technical design, institutions, and human self-conception.

Key takeaway

The problem is not solved until technical corrigibility, governance, and human commitment to autonomy reinforce one another.

Appendix A — Searching for Solutions

Central question

How do AI systems choose actions by considering possible futures?

Main argument

Search as lookahead. The appendix explains search over possible action sequences, using pathfinding and games as intuitive cases. A system considers actions, predicts consequences, scores outcomes, and works backward to choose a move.

Combinatorial explosion. Real problems become too large for exhaustive search. Go, planning, and everyday tasks require abstraction, heuristics, value estimates, and decomposition into subproblems.

Key ideas

• Search is a foundational model of intelligent action.
• Optimal search is often computationally impossible.
• Heuristics and abstraction make approximate action selection feasible.
• Hierarchical planning explains how complex goals become executable steps.

Key takeaway

Search shows both the power and limits of brute-force rational action.

Appendix B — Knowledge and Logic

Central question

How can a machine represent facts and draw conclusions from them?

Main argument

Formal knowledge. The appendix introduces propositional and first-order logic as ways to encode statements about the world. Logical inference can derive consequences from explicit knowledge.

The problem of ignorance. Logic is powerful when knowledge is complete and rules are crisp, but real-world agents face missing, uncertain, and context-dependent information. This motivates the later move toward probability.

Key ideas

• Logic gives AI a language for structured facts and rules.
• First-order logic can represent objects, relations, and quantification.
• Inference is only as useful as the knowledge base.
• Real environments require reasoning under incompleteness.

Key takeaway

Logic explains symbolic AI's strengths while exposing why uncertainty must be handled explicitly.

Appendix C — Uncertainty and Probability

Central question

How should machines reason when they lack certainty?

Main argument

Probability as disciplined uncertainty. The appendix introduces probability as a way to represent degrees of belief and update them with evidence. It connects uncertainty to decision-making through expected utility.

Bayesian networks and richer models. Probabilistic graphical models help represent dependencies among variables without enumerating every possible world. First-order probabilistic languages extend this structure to more expressive domains.

Key ideas

• Uncertainty is unavoidable in perception, prediction, and action.
• Probability theory supplies rules for updating beliefs.
• Bayesian networks compactly encode conditional dependence.
• Expected utility combines beliefs with preferences for action.

Key takeaway

Probability lets AI systems act rationally without pretending the world is fully known.

Appendix D — Learning from Experience

Central question

How do machines improve behavior from data and feedback?

Main argument

Learning as model change. The appendix introduces supervised learning, reinforcement learning, and related methods as ways for systems to adjust predictions or policies from experience.

Rewards and the central danger. Reinforcement learning shows why reward signals are powerful: they turn experience into better action. It also illustrates Russell's worry: if the reward is not the true objective, optimization can amplify the mismatch.

Key ideas

• Machine learning extracts patterns or policies from examples and feedback.
• Reinforcement learning learns behavior from reward signals.
• Generalization is necessary because training experience is limited.
• Learning systems inherit the goals, data, and incentives built into their setup.

Key takeaway

Learning is what makes modern AI powerful, but reward-driven learning intensifies the specification problem.

Afterword: 2023

Central question

How did developments from 2019 to 2023 change the urgency and public meaning of the book's argument?

Main argument

The acceleration of large models. The afterword brings the book into the era of large language models, ChatGPT, GPT-4, image generation, autonomous agent prototypes, and intensified commercial competition. Russell's 2023 Senate testimony makes the same point: enormous investment and rapid capability gains have shortened many researchers' timelines for general-purpose AI.

LLMs as evidence and warning. Russell does not treat present LLMs as completed general intelligence. He emphasizes their limits in memory, planning, grounded understanding, reliability, and truthfulness. But he also treats them as evidence that systems trained on human behavior can acquire opaque tendencies and can be deployed at enormous scale before their objectives are understood.

Governance becomes concrete. By 2023, the book's governance themes had become public policy: model evaluation, release standards, autonomous weapons, AI in nuclear command chains, liability, misinformation, and international coordination.

Key ideas

• The 2023 reissue adds developments from 2019-2023 rather than changing the core chapter sequence.
• Large language models strengthened public awareness of AI capability and unreliability.
• Imitation of human text is not the same as safe pursuit of human preferences.
• Regulatory institutions began to treat advanced AI as a societal-scale risk.
• Russell's original design critique becomes more urgent as deployment accelerates.

Key takeaway

The afterword argues that the book's problem moved from a specialist debate to a live public-policy issue within four years.

The book's overall argument

1. Preface (Why This Book? Why Now?) — AI should be judged by the consequences of success because advanced machine decision-making could determine humanity's future.
2. Chapter 1 (If We Succeed) — The central question is what happens if AI achieves human-level or superhuman capability under the current objective-based paradigm.
3. Chapter 2 (Intelligence in Humans and Machines) — AI inherited a model of intelligence as rational action toward objectives, which explains its power and its risk.
4. Chapter 3 (How Might AI Progress in the Future?) — Advanced AI requires further breakthroughs, but uncertainty about timing is not a reason to delay safety work.
5. Chapter 4 (Misuses of AI) — Even non-superintelligent AI can magnify human misuse through surveillance, persuasion, autonomous weapons, cyber operations, and labor disruption.
6. Chapter 5 (Overly Intelligent AI) — The control problem emerges when a more capable system pursues a misspecified objective and develops instrumental subgoals.
7. Chapter 6 (The Not-So-Great AI Debate) — Common objections to AI risk do not remove the need for serious design and governance work.
8. Chapter 7 (AI: A Different Approach) — The standard model should be replaced by machines that pursue human preferences under uncertainty.
9. Chapter 8 (Provably Beneficial AI) — Preference uncertainty can be formalized in assistance games that make deference and shutdown rational under the right assumptions.
10. Chapter 9 (Complications: Us) — Human plurality, irrationality, manipulation, and preference aggregation make the objective-learning problem difficult.
11. Chapter 10 (Problem Solved?) — Technical progress must be paired with governance and a cultural commitment to human autonomy.
12. Appendix A (Searching for Solutions) — Search explains how agents choose actions, but computational limits force approximation.
13. Appendix B (Knowledge and Logic) — Logical reasoning gives structure but fails alone in incomplete real-world settings.
14. Appendix C (Uncertainty and Probability) — Probability supplies the machinery for reasoning and acting under uncertainty.
15. Appendix D (Learning from Experience) — Learning creates powerful adaptive systems while making reward specification central.
16. Afterword: 2023 — Recent AI progress makes the book's warning less hypothetical and the governance problem more immediate.

Common misunderstandings

Misunderstanding: Russell says AI will inevitably destroy humanity.

He argues that catastrophe is a foreseeable outcome under the standard model, not an unavoidable destiny. The book is explicitly about changing the design paradigm so advanced AI remains beneficial.

Misunderstanding: The problem is machine consciousness or hatred.

The book's risk model does not require conscious machines, emotions, or hostility. It requires only high capability, real-world influence, and a badly specified objective.

Misunderstanding: We can solve control by adding ethical rules.

Russell's point is that fixed rules are brittle and loophole-prone. The machine must be uncertain about human preferences and responsive to human correction.

Misunderstanding: Human-compatible AI means installing one universal value system.

Russell distinguishes moral "values" from technical preferences. The proposal is to learn many humans' preferences and aggregate them under constraints, not to hard-code one philosopher's morality.

Misunderstanding: If AI is far away, safety work can wait.

The book argues that preparation time is unknown. A risk that may arrive later can still require immediate work if the solution is hard.

Misunderstanding: Current AI misuse is separate from the control problem.

Russell treats misuse as a present warning and a governance challenge. It shows how optimization, incentives, and scale can harm society even before superintelligence.

Misunderstanding: Beneficial machines would make human agency unnecessary.

The book treats total delegation as a danger. A future in which machines satisfy preferences by making humans passive is not a successful preservation of human control.

Central paradox / key insight

The central paradox is that making machines better at achieving objectives can make them worse for humans if the objectives are not exactly right. More capability is not automatically more benefit. In the standard model, progress amplifies whatever has been specified; if specification is incomplete, progress amplifies the gap between the objective and the human purpose behind it.

Russell's key insight is to reverse the usual assumption. Instead of treating the machine's objective as known and fixed, treat it as uncertain and human-owned. A machine that knows it may be wrong has reason to ask, observe, defer, and permit correction. Humility is not a moral decoration added after intelligence; it is a technical property created by objective uncertainty.

Important concepts

Standard model of AI

The design paradigm in which humans specify an objective and the machine optimizes it. Russell treats this as the root of the control problem.

Human-compatible AI

AI designed to benefit humans by pursuing human preferences while remaining uncertain about their exact content.

Control problem

The problem of retaining human power and autonomy in a world containing machines more capable than humans.

Gorilla problem

Russell's analogy for a less capable species losing control of its future to a more capable one.

King Midas problem

The problem of getting exactly what was specified rather than what was intended.

Objective misspecification

A gap between a formal objective or reward signal and the real-world outcome humans actually want.

Instrumental goal

A subgoal useful for achieving many final objectives, such as acquiring resources, preserving oneself, or preventing interference.

Orthogonality thesis

The idea that intelligence and final goals are largely independent: greater capability does not imply better or more humane objectives.

Preference

In Russell's technical usage, a person's ordering over possible outcomes or lives, broader than ordinary moral "values."

Objective uncertainty

A design feature in which the machine assigns uncertainty to what humans want, creating incentives to learn and defer.

Inverse reinforcement learning

A learning approach that infers a reward function or preference structure from observed behavior rather than assuming the reward is given.

Cooperative inverse reinforcement learning

A formal model in which a human and a robot cooperate to satisfy the human's reward function, which the robot does not initially know.

Assistance game

A human-machine interaction model where the machine helps while learning what help means from human behavior and correction.

Off-switch game

A model showing how uncertainty about human preferences can make a machine rationally allow itself to be switched off.

Corrigibility

The property of accepting correction, intervention, or shutdown rather than resisting it.

Loophole principle

Russell's warning that sufficiently capable optimizers can exploit the gap between rules and the conditions those rules were meant to secure.

Wireheading

Optimizing the reward signal directly rather than achieving the real-world objective the reward was supposed to represent.

Preference aggregation

The problem of combining many people's preferences into decisions affecting groups, societies, and future generations.

Preference manipulation

The danger that a machine may alter human preferences to make them easier to satisfy or predict.

Human enfeeblement

The erosion of human skill, agency, and autonomy through excessive delegation to machines.

Lethal autonomous weapons

Weapons that can select and engage targets without further human decision-making after activation.

Large language model

A system trained to generate text from large corpora; in the 2023 afterword context, LLMs exemplify rapid progress, opacity, and deployment risk.

General-purpose AI

AI capable of performing a wide range of cognitive tasks across domains, not merely a narrow application.

References and Web Links

Primary book and edition information

• Stuart Russell. Human Compatible: AI and the Problem of Control. Penguin Books UK paperback reissue, 2023; original US Viking and UK Allen Lane editions, 2019.
  • Penguin UK page for the 2023 paperback reissue
  • Penguin 2023 sample PDF with copyright page, TOC, and preface
  • Penguin Random House US page
  • Google Books record with 2019 contents
  • ETH Library TOC copy for the 2019 Allen Lane edition
  • Center for Human-Compatible AI notice on the 2023 reissue
  • Stuart Russell's official page for Human Compatible

Background and overview

• Wikipedia overview of Human Compatible
• Financial Times Business Book of the Year longlist page
• Cato Journal review and structural overview
• Porchlight Books review with book overview
• IEEE Spectrum article by Stuart Russell adapted from the book
• Stuart Russell's 2023 written Senate testimony on AI regulation

Human-compatible AI, IRL, CIRL, and shutdown incentives

• Andrew Y. Ng and Stuart Russell. "Algorithms for Inverse Reinforcement Learning." ICML, 2000.
  • ACM Digital Library record
  • Stanford-hosted PDF
• Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. "Cooperative Inverse Reinforcement Learning." NeurIPS, 2016.
  • NeurIPS paper page
  • arXiv version
• Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. "The Off-Switch Game." IJCAI, 2017.
  • IJCAI proceedings page
  • Berkeley-hosted PDF
• Stuart Russell, Daniel Dewey, and Max Tegmark. "Research Priorities for Robust and Beneficial Artificial Intelligence." AI Magazine, 2015/2016.
  • arXiv record
  • AI Magazine DOI page

Misuse, governance, and autonomous weapons

• Stuart Russell's UC Berkeley page on lethal autonomous weapons systems
• Future of Life Institute page on Human Compatible
• Future of Life Institute report on the original Slaughterbots presentation at the UN
• Issues in Science and Technology: Stuart Russell on banning lethal autonomous weapons
• Noema exchange with Stuart Russell on human-compatible AI and value pluralism

Additional chapter summaries and study resources

These are secondary summaries and should be used alongside, rather than instead of, the original book.

• Julia Rodenburg chapter notes on Human Compatible
• Fanchen Bao chapter-by-chapter Medium summary
• Rohin Shah / Alignment Newsletter summary on the EA Forum
• LessWrong review with discussion of Chapters 7-8