In early 2023, hundreds of AI scientists, including Nobel laureate Geoffrey Hinton and Turing Award winner Yoshua Bengio, signed a one-sentence open letter declaring that mitigating AI extinction risk should be a global priority. Eliezer Yudkowsky and Nate Soares, leaders of the Machine Intelligence Research Institute (MIRI), also signed it but considered the statement a severe understatement. Their concern centers around future artificial superintelligence (ASI), which they describe as machine intelligence surpassing humanity at nearly every mental task.

Yudkowsky founded MIRI after beginning work on machine superintelligence in 2001, and the organization became the first to focus on ensuring that superintelligent AI benefits rather than harms humanity. Yudkowsky initially sought to build superintelligence in 2000, but by 2001, he had begun to suspect that it might not reliably act in humanity’s interests, and by 2003 understood the difficulty of the alignment problem: the challenge of ensuring that advanced AI systems consistently pursue human goals and values. For two decades, MIRI conducted technical research, though the authors view some downstream effects with regret, including introducing DeepMind’s founders to their first major investor and influencing Sam Altman to start OpenAI.

As AI capabilities accelerated through major breakthroughs—such as AlexNet in 2012, AlphaGo in 2016, GPT-3 in 2020, ChatGPT in 2022, and reasoning models in 2024—and safety research lagged far behind, the authors concluded that humanity could not engineer its way to safety in time.