The chapter begins with a warning illustrated by hormone replacement therapy (HRT). Based on observational studies using regression analysis, including the Nurses’ Health Study at Harvard, millions of postmenopausal women were prescribed estrogen supplements in the 1960s to prevent heart disease. These studies found that women taking estrogen had one-third as many heart attacks as those not taking it. However, subsequent randomized clinical trials revealed that estrogen actually increased risks of heart disease, stroke, blood clots, and breast cancer. The New York Times Magazine estimated tens of thousands may have died prematurely due to this misinterpretation of observational data. However, illustrating Wheelan’s point about the complexity of statistics and the need for astute analysis rather than a reliance on precise math is the fact that after Naked Statistics was published, new research and a reexamination of the clinical trials results have concluded that HRT is actually incredibly beneficial for women, provided they begin treatment during perimenopause or immediately after; the original worrying results were skewed by older women in the study pool

The author presents seven common regression pitfalls. First, applying regression to nonlinear relationships produces meaningless results, like trying to use a single line to describe a curved pattern. Second, correlation does not establish causation; associations can be spurious or reflect reverse causality, where the supposed effect actually causes the supposed cause. Third, omitting important variables creates bias when other variables incorrectly absorb the omitted variable’s effect. For example, analyzing golf’s health effects without controlling for age falsely suggests golf causes disease, when golfers simply tend to be older.

Fourth, including highly correlated explanatory variables (multicollinearity) prevents the analysis from distinguishing their separate effects. Fifth, extrapolating beyond the sample population produces invalid predictions, like using adult data to predict a newborn’s weight. Sixth, including too many variables risks finding statistically significant results by chance alone—data mining that yields false discoveries.

Even peer-reviewed medical research frequently fails to replicate. Researcher John Ioannidis estimates that roughly half of published scientific papers are eventually proven wrong. This emphasizes two lessons: Properly specifying which variables to include matters more than calculations, and regression builds only a circumstantial case requiring theoretical justification and replication.

Program evaluation seeks to measure an intervention’s causal effect by comparing its outcomes with the counterfactual, or what would have happened without the intervention. The challenge of properly evaluating programs is illustrated by studying whether more police reduce crime. Simply comparing jurisdictions with different police levels fails because cities differ fundamentally. Even regression analysis encounters reverse causality problems, as high-crime areas hire more officers.

Jonathan Klick and Alexander Tabarrok addressed this by exploiting the Washington, DC terrorism alert system as a natural experiment. On high-alert days, DC deploys additional police for terrorism concerns unrelated to street crime. The researchers found that ordinary crime dropped roughly 7% on high-alert days, demonstrating that police presence deters crime.

The chapter outlines five program evaluation approaches. Randomized, controlled experiments represent the gold standard; these randomly assign subjects to treatment and control groups. Examples include a study finding that prayers by strangers do not improve heart surgery outcomes, and Tennessee’s Project STAR, which demonstrated that smaller class sizes enhance student learning. 

However, it is often logistically or ethically impossible to create randomized controlled experiments. For this reason, the second approach is natural experiments, which allow researchers to exploit (in this case a technical term that just means to use data from) circumstances that happened to create treatment and control groups accidentally. For example, Adriana Lleras-Muney noticed that states passed laws requiring minimum years of schooling at different rates. She examined these historical variations to show that one additional year of education extended life expectancy by 18 months.

Third are nonequivalent controls, which use nonrandomized but similar comparison groups. Stacy Dale and Alan Krueger compared students who attended elite colleges with equally talented students who had been accepted to such schools but ended up attending less selective institutions instead. They found no significant earnings differences after college for all except low-income students.

Fourth, difference-in-differences compares how outcomes change over time between treatment and control groups, isolating the intervention’s effect from broader trends. Finally, Discontinuity analysis compares groups just above and below arbitrary eligibility cutoffs. Randi Hjalmarsson used this approach with Washington State’s juvenile sentencing guidelines, finding that incarceration deterred future crime.

The chapter concludes that careful methodology enables researchers to approximate counterfactuals and identify causal relationships even when controlled experiments are impossible.

The conclusion notes that society is now overwhelmed with data, contrasting sharply with earlier eras when policymakers worked blind. During the Great Depression, Herbert Hoover declared recovery in 1930 based on faulty data showing 2.5 million unemployed, when actually 5 million were jobless and unemployment was climbing by 100,000 weekly.

Five socially significant questions demonstrate statistics’ power. The first concerns the future of football. Mounting evidence links the sport to chronic traumatic encephalopathy and other long-term brain damage. Researchers are using statistical tools to determine whether football can be played safely or whether, as author Malcolm Gladwell suggested, it resembles dog fighting: An activity that was seen as acceptable in the past but is now viewed as willingly subjecting participants to suffering.

The second question addresses autism spectrum disorder diagnoses, which have nearly doubled in a decade, with the condition affecting 1 in 88 children in 2012, when Wheelan was writing. (In 2023, 1 in 36 children received this diagnosis). Statistical analysis seeks to determine whether this represents a true “epidemic” or merely increased awareness, while attempting to identify causal factors. Statistics has already debunked false associations with vaccines, which definitively do not cause autism.

Third is assessing our educational system. Value-added models use student test score gains to evaluate teachers, but these assessments contain substantial statistical noise. Economist Doug Staiger warns that the correlation in year-to-year teacher performance is only 0.35, similar to baseball players’ performance correlation. Air Force Academy data reveal that professors producing impressive immediate test scores may actually hinder long-term learning compared to experienced instructors who emphasize fundamental concepts.

Fourth has to do with global poverty. Esther Duflo has helped reshape development economics by conducting randomized experiments in low-income countries. Her work has demonstrated that interventions like subsidized fertilizer delivery and photographic attendance monitoring can significantly improve outcomes. A natural experiment in Côte d’Ivoire also revealed that when women earned extra income, they spent it on family food, whereas men were less likely to do so.

Finally, there is the question of data mining and privacy. The marriage of data and technology creates unprecedented privacy challenges. Target’s predictive analytics identified pregnant shoppers from their purchasing patterns, sometimes before family members knew. The US Supreme Court ruled that law enforcement cannot track vehicles without warrants.

The book concludes that statistics is like fire or knives—an immensely useful tool that is dangerous when misused. Data must be employed wisely, with human judgment always accompanying mathematical analysis.