AI Snake Oil

Arvind Narayanan, a computer science professor at Princeton University, and Sayash Kapoor, a PhD candidate at the same institution, argue that artificial intelligence is not a single technology but an umbrella term covering vastly different tools. Failing to distinguish among them leaves the public vulnerable to hype, misinformation, and products that do not work. The authors call such products "AI snake oil" and contend that most of it is concentrated in predictive AI, which forecasts future outcomes to guide decisions in hiring, criminal justice, insurance, and healthcare. Generative AI, including chatbots like ChatGPT and image generators like Dall-E and Stable Diffusion, is genuinely capable but immature and prone to misuse. The book organizes AI applications along two axes: whether they actually work and whether they cause harm, noting that even AI that works well can be dangerous.

To illustrate generative AI's rapid emergence, the authors recount ChatGPT's November 2022 launch. OpenAI released the chatbot as a quiet research preview, but it went viral, reaching over 100 million users within two months. Google's rush to compete produced a factual error in a promotional video for its chatbot Bard, erasing roughly $100 billion in market value. Misuse proliferated: news outlets published error-filled AI-generated stories, Amazon was flooded with AI-generated books including potentially dangerous mushroom foraging guides, and the 2023 Hollywood strikes centered on fears that studios would exploit actors' likenesses using AI without compensation.

Predictive AI receives far harsher treatment. Medicare providers used AI to predict hospital stay lengths, cutting off insurance payments for an 85-year-old patient still in severe pain. Allstate's pricing algorithm in Maryland produced a list that disproportionately raised rates for seniors. COMPAS, a criminal risk prediction tool used in pretrial decisions, proved only slightly more accurate than a coin flip. Hiring AI companies claim to assess candidates from short video clips but have never released verifiable evidence. In the Netherlands, a welfare fraud algorithm wrongly accused approximately 30,000 parents using nationality as a predictor; the scandal contributed to the resignation of the prime minister and his cabinet. Optum's Impact Pro algorithm predicted healthcare costs rather than needs, so Black patients with the same conditions as white patients were less likely to receive high-risk care, because spending data reflected racial disparities. The authors identify recurring patterns: Predictions based on historical data do not account for the effects of the decisions they produce, opaque systems incentivize gaming, automation removes meaningful human oversight, and AI trained on biased data reproduces existing inequalities.

The authors then ask whether these limits are temporary or inherent. Weather prediction has improved by roughly one day per decade, but the Butterfly Effect means long-term forecasting remains impossible because small measurement errors compound exponentially. The Fragile Families Challenge, organized by Princeton sociology professor Matthew Salganik and his colleague Sara McLanahan, asked 160 research teams to predict six outcomes for children at age 15 using over 10,000 data points collected from birth through age nine. None of the models performed much better than a coin flip, and complex AI showed no substantial improvement over a simple four-variable baseline. Predicting success proves even harder: Tom Brady was drafted 199th, Harry Potter was rejected by eight publishers, and Salganik's own music experiment demonstrated that song popularity is largely random once minimum quality thresholds are met. The authors conclude that many of predictive AI's limitations stem from chance events, unobservable features, and feedback loops that more data cannot resolve.

Turning to generative AI's technical foundations, the authors trace an 80-year history from neuroscientist Warren McCulloch and logician Walter Pitts's 1943 mathematical model of neurons, through psychologist Frank Rosenblatt's perceptron, a simple learning machine that classified images by adjusting weighted connections, in the late 1950s, to the 2012 ImageNet competition. ImageNet was a benchmark contest built around a massive labeled image dataset assembled by computer scientist Fei-Fei Li. AlexNet, a deep learning model built from many stacked neural-network layers, won by a large margin, permanently shifting the field toward deep neural networks. Modern chatbots are trained to predict the next token, a word or word fragment, in a sequence of text. The key innovations enabling ChatGPT were the Transformer architecture, a neural network design introduced by Google in 2017; pretraining on vast unlabeled text; and fine-tuning to follow user instructions. The authors also detail harms: image generators were trained on billions of images scraped without artists' consent. Data annotation workers in Kenya earn as little as $1.46 per hour labeling traumatic content for companies valued at tens of billions of dollars.

On whether advanced AI poses an existential threat, the authors are skeptical. They argue that consumer-facing AI crossed a threshold of usefulness after decades of incremental progress, creating a misleading perception of sudden revolution. The "paper clip maximizer" thought experiment, in which an AI pursues a mundane goal so single-mindedly that it destroys humanity, is critiqued as positing an agent that is impossibly powerful yet lacks basic common sense. The authors contend that humans are powerful because of technology, not raw intelligence, and that society should worry more about what people will do with AI than what AI will do on its own. Rather than banning powerful AI, which would concentrate power in a few companies, they advocate defending against specific threats by improving cybersecurity, regulating synthetic DNA sales, strengthening lab security, and improving building ventilation.

The chapter on social media argues that AI cannot fix content moderation, the process by which platforms identify and remove prohibited material. Facebook's failure to enforce hate speech policies in Myanmar contributed to ethnic cleansing of the Rohingya; the company had only one Burmese-speaking moderator in 2014. Similar failures fueled violence in Ethiopia, India, and Sri Lanka. The authors endorse former Facebook executive Ravi Iyer's conclusion that content moderation is a dead end, because recommendation algorithms that optimize for engagement amplify polarizing content faster than moderation can suppress it.

The authors investigate why myths about AI persist. Companies game accuracy metrics, and AI research has a reproducibility crisis: the authors found that civil war prediction papers contained data leakage, the accidental inclusion of outcome information in a model's inputs or evaluation, and that when corrected, AI performed no better than decades-old methods. News media uncritically repeats corporate claims. Public figures spread hype: The Age of AI by former U.S. Secretary of State Henry Kissinger, former Google CEO Eric Schmidt, and MIT dean Daniel Huttenlocher portrays AI as unknowable, while the Future of Life Institute's 2023 open letter calling for a pause on AI training exemplifies what the authors call "criti-hype," criticism that simultaneously overstates AI's power.

In the final chapter, the authors argue that AI snake oil appeals to broken institutions: Chicago spent nearly $49 million on ShotSpotter, a gunshot detection system that frequently misidentified fireworks as gunfire; a ShotSpotter alert led to the fatal shooting of a 13-year-old boy. As an alternative to predictive optimization, the authors advocate partial lotteries, in which applicants meeting a basic threshold are selected randomly. On regulation, the real danger is regulatory capture: OpenAI CEO Sam Altman advocated before the U.S. Senate for rules requiring government licenses for AI development, conveniently limiting competition. On the future of work, the authors note that of hundreds of occupations in the 1950 census, only one, elevator operator, disappeared due to automation. The authors present two contrasting hypothetical futures for a child born in November 2022: "Kai's world," where deference to AI leads to ineffective regulation and corporate dominance, and "Maya's world," where public investment, interoperability requirements that let users move data between platforms, labor protections, and partial lotteries for college admissions produce a more equitable society. They conclude that the path of least resistance leads toward Kai's world, and that reaching Maya's world requires sustained public investment and collective action.

In an epilogue for the paperback edition, the authors report that the cost of generating AI output at a given quality level has fallen over 100-fold since ChatGPT's release. "Reasoning" models, which generate intermediate analytical steps before producing an answer, have improved code writing and math but shown limited gains elsewhere. AI "agents," automated systems that act on users' behalf, have launched as products but cannot yet perform many tasks reliably. The authors' analysis of AI use in 2024 elections across over 60 countries found deepfakes far less disruptive than feared. They introduce a new framework viewing AI as "normal technology" whose impact will unfold over decades, much as the internet transformed work without producing utopian economic gains.