What the Credibility Revolution in Empirical Economics Means for Legal Analysis

Professor Dhammika Dharmapala speaks at a lectern

Empirical economics’ recent surge in credibility—a movement spurred by better research designs and the use of causal inference—is transforming law and economics from a largely theoretical field to a maturing one that uses randomized field experiments, quasi-experiments, and other methods to test the impact of legal changes, a shift that could have implications for legal practice and policymaking, Professor Dhammika Dharmapala said in this year’s Ronald H. Coase Lecture in Law and Economics.

“If we can provide more credible estimates of empirical effects, and do so in a more unified way, then perhaps policymakers would listen more carefully,” said Dharmapala, the Julius Kreeger Professor of Law. “And for practitioners, it may tell them something concrete about the world that is relevant to the law.”

In his talk, Dharmapala traced the evolution of empirical design in law-and-economics scholarship through papers on two topics—the value of securities regulation and the deterrent effect of capital punishment—showing how improvements in empirical design, especially the use of control groups, had allowed scholars to make more credible claims about causal relationships. Increasingly, law-and-economics scholars have used randomized experiments, analogous to the randomized trials used in medicine, and quasi-experiments, which allow researchers to estimate causal impacts without using randomized assignments to treatment or control groups. The use of these methods represents a fundamental “change in focus from statistical theory to looking carefully at variation in the real world,” Dharmapala said—and the result has been a “credibility revolution,” a term borrowed from a 2010 paper about empirical advances in economics as a whole.

“Recent developments in empirical scholarship have greatly enhanced the credibility of the findings of empirical law and economics scholars,” Dharmapala said. “This is important for law and economics itself, but law and economics also plays a very important role in this credibility revolution because so many of the quasi-experiments come from variations in the law.”

Contributing to this shift has been the rise of big data sets, as well as the increased sharing of such data, Dharmapala said.

“It used to be the case that you’d publish a finding and the dataset wouldn’t be available to anyone else,” he said. “But today journals are increasingly requesting or even demanding that those who publish in a journal make their datasets readily available to those who wish to replicate their findings.”

The talk inspired lively discussion among faculty members, some of whom debated whether the search for proper research conditions, including the ability to designate both control and treatment groups in a quasi-experiment, meant that scholars were focusing their research too narrowly.

“We might end up chasing increasingly idiosyncratic or bizarre research designs, focusing on things that are weird instead of things that are important,” Assistant Professor Adam Chilton said. “Or we could increasingly work with policymakers to create our own variation.”

Dharmapala agreed that it can seem like a tough dilemma between “saying something very credible about an idiosyncratic setting as opposed to saying something rather incredible about a more important question,” adding that empirical law-and-economics scholars are likely to increasingly look for opportunities to create their own variation.

Omri Ben-Shahar, the Leo and Eileen Herzel Professor of Law and the Kearney Director of the Coase-Sandor Institute for Law and Economics, said that empirical research still suffers from credibility issues, noting that results can differ from study to study and that there is a danger that research will simply end up confirming a scholar’s prior belief or underlying politics. (With this concern in mind, he and Chilton last fall circulated the draft of a paper on the design of privacy disclosures before they had collected the data, which measured the impact on respondents’ behavior).

Dharmapala agreed that all of these concerns were significant, but argued that the shift nonetheless represents progress.

“Your concern about ideology or politics would be even more serious in the absence of this credibility revolution,” Dharmapala said to Ben-Shahar. “If we didn’t have widespread agreement as to what (qualified as) a quasi-experiment then people would have even greater leeway to find results consistent with their priors. Having said that, what the credibility revolution does is provide a way to decouple the ideology of researchers from their findings, at least in combination with other developments, such as the posting of data online.”