New Statistical Tool: Did That Law Actually Work?
Better answers about whether policies make a difference — for everyone at once.
How do we know if a law or policy actually changed anything? Normally, you’d compare a group affected by the policy to a similar group that wasn’t. But sometimes a policy affects everyone at once — like a nationwide gun law or a Supreme Court decision. There’s no untouched “control group” to compare against.
This new paper offers a clever solution. Instead of looking for a comparison group, it studies how things were trending before the policy and uses that history to predict what would have happened afterward. But it doesn’t just assume one single prediction. The method, built on something called Gaussian processes (a type of machine learning), keeps many possible “future without the policy” curves. It then shows a range of possibilities that only grows wider the further you look into the future — honestly showing that you can’t predict forever with confidence.
This matters because older methods often gave overly confident answers, acting like they knew exactly what would have happened. That can mislead policymakers, courts, and the public. The new tool makes uncertainty visible, so you know when a measured effect is reliable and when it’s just guesswork.
The author tested the method using handgun purchases after the 2008 Supreme Court decision Heller v. D.C., which struck down a handgun ban. The analysis shows the method working in a real-world case where everyone was affected nationally. An R package called gpss is available for other researchers and analysts to use. In short, this is a step toward more truthful answers about what policies truly accomplish.
- This new method measures a policy’s effect when everyone is affected at once, so there’s no control group to compare with.
- It uses machine learning (Gaussian processes) to forecast what would have happened without the policy, while showing how uncertain that forecast is.
- The test case — handgun purchases after the Heller decision — shows it can reveal real-world policy effects more honestly than older methods.
Why It Matters
Gives citizens and policymakers more honest evidence about whether laws and rulings actually work, not just confident guesses.