Randomization is a key ingredient of rigorous causal inference. Given the increasing desire for rigorous evaluation of the effectiveness of government programs, there are excellent reasons to start designing public policies with randomized implementation plans from the outset. But convincing government agencies to randomize policies does not necessarily ensure that researchers will reach consensus about their effectiveness. After all, randomization is not the only ingredient of good causal inference.
A major concern when randomizing public policy implementation is the risk of multiple hypothesis testing. If not properly accounted for, this problem could result in expensive, controversial efforts that produce little more than a morass of inconclusive results.
Simply put, multiple hypothesis testing occurs when researchers use the same data to test many hypotheses simultaneously—or if not simultaneously, at least separately. The problem with doing this is that conventional tests for statistical significance are more likely to produce false positives when dozens of hypotheses are tested using the same data.
Read more at The Regulatory Review