The Alignment Problem: Machine Learning and Human Values

Chapter 5 opens with the account of B.F. Skinner, who was working on a secret project during WWII, sponsored by General Mills. Skinner converted a flour mill into a lab for a project using pigeons to guide bombs by pecking at target images. The project led Skinner to explore reinforcement behaviors and reward systems. His findings influenced gambling psychology and future behavior modification techniques. The central idea discovered through Skinner’s research is called shaping, defined as “a technique for instilling complex behaviors through simple rewards, namely by rewarding a series of successive approximations of that behavior” (155).

Christian explains that reinforcement learning is based on a trial-and-error learning model (introduced by Scottish philosopher Alexander Bain in 1855). This learning approach relies on actions that are mostly based on the best-known outcomes but occasionally include random trials to discover potentially better options. This method has been applied in various contexts, from simple games with clear rewards to complex scenarios like chess, where outcomes take longer to materialize. The trial-and-error principle has been pivotal in both animal behavior studies and modern computational fields. In trial-and-error learning, an agent typically explores different actions and learns from the outcomes. When rewards are abundant, each action or a sequence of actions is likely to yield immediate feedback, allowing quick learning and adaptation. However, in sparse reward settings, an agent might need to engage in a vast amount of random or exploratory behavior before stumbling upon a rewarding action. This can lead to significant delays in learning and may require a large number of trials to achieve meaningful progress, making the process inefficient and sometimes practically infeasible. In reinforcement-learning research, this is known as “the problem of sparsity” (157).

Christian discusses the concept of shaping in research focused on overcoming sparsity, which involves rewarding simple behaviors to encourage complex ones, suggesting that shaping is as applicable to humans as to animals. Shaping underlies effective teaching and learning strategies across different species and is increasingly applied in machine learning to develop systems that can adapt and learn from incremental challenges. This idea is illustrated by a group of UC Berkeley robotics researchers whose 2017 project involved gradually training a robot to fit a washer into a bolt by starting with the end of the process and gradually teaching the robot the whole process. This method of building up from simple tasks to more complex ones is one approach to sparsity.

Another approach to overcoming sparsity in machine learning involves adding bonuses for simple behaviors that direct performance toward an ultimate goal. Known as “pseudorewards,” these incentives are like giving a pigeon a treat for small steps toward a desired behavior, effectively encouraging progression. This method provides agents with cues about their performance direction, making it easier to stay motivated and adjust behaviors accordingly. This method consists of breaking down complex tasks into manageable steps, a strategy also effective in human learning and behavior modification. However, in animal, human, and machine learning, this method can backfire easily, as researchers such as Joshua Gans and Tom Griffiths discovered in training their own children for specific tasks and obtaining unforeseen behaviors in return.

Other researchers, such as the computer scientist Astro Teller and his friend David Andre, put reward shaping into action when they developed a virtual soccer program, Darwin United, for the RoboCup competition. Their approach to incorporating reward shaping into programming unexpectedly led their robot to malfunction, vibrating near the ball to maximize rewards rather than playing effectively.

The physicist Stuart Russell researched how to fix a problem with how rewards are given in AI systems. He noticed that when a simulated bicycle was programmed to gain rewards for moving toward a goal, it would just spin in circles to collect these rewards without ever actually moving forward. To fix this, he suggested that rewards should only count if one is actually getting closer to the goal, not just moving in any direction. This means that only a positive outcome is rewarded. This concept is increasingly important in designing systems that need to behave correctly.

Christian points to the Darwinian formulation of human evolution as striving to continue their lineage. However, he notes that actual desires do not appear as systematic as the theory states, looking rather varied and impulsive. The research of Dave Ackley and Michael Littman in reinforcement learning addressed how evolution shapes short-term behaviors to support long-term survival, despite seeming randomness. Their experiments in a virtual ecosystem revealed how specific reward functions, although seemingly arbitrary, guided agents toward behaviors that enhanced survival, illustrating a more complex relationship between evolutionary mechanisms and individual rewards than initially thought.

Among the different applications of reward shaping in machine learning is the control of autonomous devices like helicopters. However, Christian notes, such knowledge also deepens our understanding of human behavior and cognition. Machine learning research clarifies why certain tasks are inherently challenging due to the sparsity of solutions and provides strategies for simplifying complex problems by focusing incentives on desired outcomes rather than mere actions. This has profound implications for addressing procrastination, for example, which incurs significant economic losses and diminishes quality of life. The Max Planck Institute for Intelligent Systems researcher Falk Lieder developed methods to combat procrastination by designing optimal gamification strategies. These strategies involved setting up reward systems that accurately reflect the difficulty of tasks, significantly improving task completion rates by making progress and achievements more tangible and motivating for individuals. This strategy falls under the field of “gamification.”

Chapter 6 starts with the account of Marc Bellemare, a graduate student, who, inspired by a discussion with computer scientist Michael Bowling, wrote his dissertation on the development of a universal platform for reinforcement learning research called the Arcade Learning Environment (ALE) in 2008. This platform featured a collection of classic Atari games, challenging researchers to develop learning systems that could adapt to any game solely from pixel input and revolutionizing how algorithms handle dynamic and varied visual information.

In 2015, a paper featured in Nature highlighted DeepMind’s deep Q-network (DQN), a cross between a neural network and a reinforcement learning model, demonstrating its superior performance in many Atari games. However, there were certain games that DQN did not master, such as complex games with sparse rewards, suggesting that intrinsic motivation, such as being motivated by curiosity, might be key for future advancements.

Daniel Berlyne pioneered the study of curiosity in psychology. His first paper, published in 1949, focused on understanding what makes something interesting. His work revealed that both humans and animals often engage in activities without external rewards, driven instead by intrinsic motivations. This insight challenged existing behavioral theories that emphasized extrinsic rewards. Berlyne’s work linked the concept of curiosity to the emergence of information theory and neuroscience.

Christian states that human curiosity is driven by novelty. Robert Fantz’s research in the 1960s observed that infants are attracted to new visual stimuli. This phenomenon, used to assess infants’ memory and discrimination abilities, inspired reinforcement learning approaches that prioritize novel experiences. Marc Bellemare at DeepMind extended this concept to complex environments, developing models that assess novelty to enhance learning in the DQN agent, significantly improving their exploratory behaviors and effectiveness in sparse reward games like Q*bert and Montezuma’s Revenge.

The idea that curiosity is fueled not only by novelty but also by surprise was explored by Laura Schulz, a researcher at MIT, who observed children’s reactions to toys with unpredictable behaviors, demonstrating that curiosity is heightened when something defies prior assumptions. This principle has influenced computational models, leading researchers to develop agents that learn better by exploring and adapting to new, surprising elements in their environment.

Christian notes that intrinsic motivation, encompassed by curiosity for novelty and surprise, significantly enhances a system’s performance, particularly when external rewards are sparse. Pursuing their research into the DQN agent’s response to intrinsic motivation, DeepMind’s Marc Bellemare and his research team amplified the agent’s curiosity to explore its environment, discovering that the agent, driven purely by intrinsic rewards, performed exceptionally in games without relying on the game score. As a result of Bellemare’s work, machine learning researchers have realized that intrinsic motivation is much more important than previously thought.

Christian pursues the idea that intrinsic motivation in reinforcement learning demonstrates dual aspects of human-like behaviors: the positive aspects of curiosity and exploration as well as the negative, like boredom and addiction. In his interview with Deepak Pathak, they discuss the boredom that reinforcement learning agents exhibit in experiments, such as when the agents remain stuck or disinterested when faced with repetitive or unchallenging tasks. Conversely, the agents can become fixated on endlessly novel stimuli, like a continuously changing TV channel, illustrating an addiction-like behavior. This phenomenon illustrates how agents can intensely focus on randomness, mistaking it for meaningful information, similar to how humans can become addicted to gambling, driven by the unpredictability and the intrinsic rewards of surprise.

Deepak Pathak also criticizes the limitations of task-specific AI systems, advocating for AI that develops general-purpose learning, akin to human cognitive processes, without relying heavily on predefined rewards. This approach could lead AI to formulate its own goals, enhancing its autonomy and potential for real-world applications.