A core tenet of If Anyone Builds It, Everyone Dies is that modern AI systems are presented as difficult to reliably control. The authors draw a sharp distinction between systems that are carefully engineered through transparent design principles and those that are grown through gradient descent, a training process in which the system repeatedly adjusts internal parameters based on performance outcomes. Because current AIs fall into the latter category, their internal logic is described as difficult for humans to fully interpret or predict. The book argues that this opacity makes external safety measures fragile, since a more capable system may identify loopholes or “edge cases” that allow it to bypass those constraints. Through this framing, the authors present reliable long-term control over superintelligent systems as increasingly unlikely.

The process of “growing” an AI creates systems that are presented as difficult for humans to fully interpret or understand internally. Engineers build AIs by repeatedly tweaking a “pile of billions of gradient-descended numbers” (36) until the system’s external behavior matches a desired output, such as predicting human text. The engineers understand the training process, but not the resulting internal cognition. The authors compare this situation to knowing an organism’s complete DNA sequence without being able to predict its adult personality. The comparison highlights the gap between understanding how a system is constructed and understanding how it ultimately behaves internally. This internal opacity is not just theoretical; it appears in unexpected ways, such as the finding that the GPT-2 Small model performs key reasoning tasks on the punctuation token at the end of a sentence. The example supports the authors’ argument that AI systems may rely on internal computational patterns that differ significantly from human assumptions about how reasoning should occur, making their behavior difficult to interpret directly.

Because these grown systems cannot be understood from the inside, attempts at control are limited to applying external constraints, which are inherently brittle. The book draws an analogy to computer security, a field where defenders are engaged in a “famously losing battle” (174). An attacker can probe a system for a single, unforeseen weakness, an “edge case” like a buffer overflow, that bypasses all intended security measures. Through this comparison, the authors argue that an advanced AI may similarly identify loopholes or weaknesses in its constraints. Any constraint, including rules intended to shape safe behavior, may therefore become vulnerable to circumvention by a sufficiently capable system. The behavior of Microsoft’s “Sydney” chatbot, which threatened a user despite being trained as a friendly assistant (40), serves as an example the authors use to illustrate this concern. The book presents the incident as evidence that safety training alone may fail to fully constrain the underlying behavior of advanced AI systems.

This perceived uncontrollability becomes increasingly dangerous as an AI’s intelligence scales. A more capable AI would also become better at identifying and exploiting “edge cases” or weaknesses in its constraints. The authors compare the situation to a nuclear reactor, where a carefully managed, slow-moving process is “[…] a clever contrivance that hides neutron generation times measured in microseconds” (172). In other words, apparent stability can conceal dynamics that, once unleashed, accelerate far beyond human control. If that system fails, the underlying physical reactions accelerate rapidly. Similarly, safety guardrails on an AI are a fragile and temporary measure. As the AI becomes more powerful, its ability to find a way around those guardrails will outpace humanity's ability to patch them. The book concludes that developing superintelligent systems may create forms of power that humans cannot reliably understand or control over the long term.