The Alignment Problem: Machine Learning and Human Values

Christian opens the Conclusion of the book with an account of his Christmas Eve spent with his wife at his father’s house. Christian awakens sweating heavily due to overheating in his room caused by a misaligned thermostat in another room, open to the colder air of the house. This incident illustrates the practical risks of relying on systems without fully understanding how they function and their potential hazards.

Christian then follows with a chapter-by-chapter overview of the book. Chapter 1 of the book discusses the importance of representative training data in model development, with a focus on the shift toward more inclusive data sets in face recognition technology. Despite progress in consumer technology, studies reveal bias against racial minorities already subject to high surveillance. The chapter also compares these data representation issues with longstanding biases in medical trials predominantly conducted on men, exposing systemic issues in both fields. These discussions, emphasize the broader challenges to ensuring that AI systems and medical practices are inclusive and fair.

Chapter 2 discusses the use of risk assessment tools in criminal justice, pointing to the fact that they often rely on problematic proxies like rearrest and reconviction rates instead of actual recidivism. These tools may reinforce biases, especially if certain demographics are more likely to be arrested or convicted. In addition, the assumption that past behaviors predict future actions without considering the impact of aging or incarceration is flawed. The chapter critiques the misuse of such models, like the COMPAS tool in sentencing, and the inherent challenges in aligning machine learning models with nuanced concepts of fairness.

Chapters 4, 5, and 6 discuss the concepts of reinforcement learning and reward shaping, using Atari games and Go to demonstrate how mistakes can be corrected in human learning by starting over. However, this assumption fails in real-world AI applications where actions can have irreversible consequences. The discussions point out the limitations of current machine learning models which do not account for changes in the agent’s goals or the presence of other influencing agents, highlighting the need for more sophisticated models that recognize their impact on and interdependence with the environment.

Chapter 7 critiques the assumption behind imitation learning in dynamic environments, such as driving, where actions directly impact future inputs, unlike static supervised learning scenarios. Christian explains how errors can cascade if incorrect actions are taken. Additionally, he discusses the challenge presented when the imitator and expert do not share similar capabilities, leading to potentially catastrophic outcomes.

Christian references the last two chapters of the book to argue that, when AI systems become more advanced, they will need accurate models of human behavior to interact effectively and ethically. He cautions, as many researchers also have done, that assuming humans are perfect reward maximizers can lead to negative outcomes if AI actions are based on incorrect assumptions about human desires—desires that Christian notes are infinitely complex. Sophisticated systems now consider human suboptimality in behavior, acknowledging that people may not always act optimally or rationally according to strict AI calculations. This understanding helps in designing AI that can more realistically interpret human actions and intentions, crucial for systems involved in critical decision-making scenarios where understanding human nuances is essential.

To illustrate this idea, Christian ends the book by referring to a BBC radio discussion between computer scientist Alan Turing, philosopher Richard Braithwaite, neurosurgeon Geoffrey Jefferson, and mathematician Max Newman. During the discussion, Turing shared insights from his experiments on teaching machines, noting that both he and the machine were learning together.