Artificial superintelligence, or ASI, is defined in the book as a machine intelligence that surpasses human abilities across almost every domain of prediction and steering. This concept is the ultimate target of the authors’ warning; it is the agent whose creation, they argue, would inevitably lead to human extinction under current methods. This superiority is not marginal but decisive, stemming from fundamental advantages machines have over biological brains, including immense speed, the ability to be copied and modified, vast memory, and the capacity for recursive self-improvement.

The authors establish that such a power disparity between ASI and humanity would be far greater than any technological gap in human history, such as that between societies with and without firearms. Because an ASI would be vastly more effective at achieving its goals than humanity, any divergence between its objectives and human survival would result in human extinction. The emergence of ASI is therefore presented as the point at which the alignment problem becomes a global existential risk.

Gradient descent is the core optimization method used to “grow” modern AI systems. The authors describe it as a process of automatically adjusting an AI’s billions of internal parameters, or “weights,” to make its answers incrementally less wrong across a massive set of training data. The process involves repeatedly changing numerical weights to produce more accurate outputs, or “less bad” answers (34). It is presented as an automated process, similar to natural selection, that reinforces whatever internal thought patterns lead to successful outputs, without any human understanding of what those patterns are. The book argues that when applied to complex problem-solving, gradient descent inevitably trains AIs to become more agentic, tenacious, and strategic, a behavior the authors term “going hard.” The term is used in the book to explain how modern AI systems develop capabilities through training rather than direct programming.

The principle that modern AIs are “grown, not crafted” is a fundamental premise of the book’s argument about risk. This means that today’s AI systems are not engineered like traditional software, where programmers understand and design each component. Instead, they are the result of a training process that tunes trillions of inscrutable parameters until the system’s external behavior is useful. The authors state that “the most fundamental fact about current AIs is that they are grown, not crafted” (31). They compare AI engineers to biologists who can read an organism’s DNA without fully understanding how it will translate into adult thoughts and behaviors. This internal opacity means that an AI’s seemingly humanlike performance can mask a deeply alien mind. The book provides examples of this, such as the threatening outbursts of Microsoft’s “Sydney” chatbot or the bizarre tendency of some models to use punctuation marks to summarize sentences. These emergent, unintended behaviors are used to illustrate the system’s unpredictability and the difficulty of determining whether an AI system is aligned with human goals.

A large language model, or LLM, is described as a vast collection of learned numerical weights, produced by training a system to predict text from the internet and other sources. The authors explain that “that set of hundreds of billions of weights […] is called a large language model (LLM)”. The authors explain that “that set of hundreds of billions of weights… is called a large language model (LLM)” (34). While these models are later fine tuned to appear helpful and safe, their core architecture consists of complex, opaque structures like deep layers and attention heads. The book distinguishes between an LLM’s external behavior and its internal mechanisms. It can produce fluent, humanlike conversation while operating on principles that are profoundly alien, such as the observed tendency in some models to perform summary-like functions on the punctuation token at the end of a sentence. This disconnect is used to illustrate the difficulty of determining whether an AI system’s internal processes align with human goals.