Unsupported claim of the day: “economics is not an experimental science”

Adam is optimistic about the spread of RCTs into economics, and (like most people in development) I share his optimism. But there is a strong resistance to the rise of experimental and quasi-experimental methods in economics. In development we mainly argue over RCTs, but in economics as a whole the focus is on “natural” experiments, where circumstances generate effectively random variation in a causal variable. One of my favorite examples of this is Doug Almond’s paper on the effect of maternal fasting during early pregnancy on a child’s health and economic outcomes, which relies in part on data from right here in Michigan. His analysis relies on the fact that fasting during Ramadan is not required for pregnant women, but it’s common for women not to realize they are pregnant early on, and fast anyway.

Recent economics Nobel-winner Chris Sims is not a fan of natural experiments in economics; indeed, he won the prize in part for his work on vector autoregression models, which typify the model-everything, nothing-is-exogenous school of thought. This short comment is a pretty accessible overview of what he doesn’t like about the quasi-experimental approach, and focuses mostly on papers that analyze the deterrent effect of the death penalty. I highly recommend it for applied stat-heads like myself.

However, I was pretty let down by Sims’ complete failure to back up his central claim: “Because we are not an experimental science, we face difficult problems of inference.” We’re not? Says who, and why? Claiming that economics is not an experimental science seems to imply that we can learn nothing from experiments, which I think is obviously false. Perhaps by that statement he is implying the weaker claim that there are some important things economists cannot learn from experiments – but by that standard medicine, with its utter failure to run an RCT on the effect of smoking on health, is not an experimental science either.

Hat tip: Dan Hirschman

Advertisements

About Jason Kerwin

http://nonparibus.wordpress.com/author
This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Unsupported claim of the day: “economics is not an experimental science”

  1. Two thoughts. First, have you read this BMJ criticism of evidence-based medicine: Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials? While the piece itself is funny, it nicely bring up the larger question of when we need experiments to establish causality for practical purposes. It was interesting that Sims mostly ignores the RCT movement, instead focusing on so-called natural experiments and various statistical “tricks” that produce quasi-experiments. I wonder if that’s because his target was A&P and their followers, and not so much the RCT movement (which will doubtless never have much impact on macroeconomics, for example)?

    Second, I think you are taking his observation to a ridiculous extreme when you write: “Claiming that economics is not an experimental science seems to imply that we can learn nothing from experiments, which I think is obviously false.” There is a rather large set of important economic problems which don’t have an easy experiment that you could attach to them, but which are still core topics in the field and which economists are still paid to understand. That should be enough to justify the claim that econ is not an experimental science. But I don’t remember him saying the stronger claim you jump to – in fact, he even argues that the quasiexperimental methods he criticizes are useful and can tell us things.. just that they are not substitutes for understanding and modeling the process that produced the data (e.g. how exactly did principals deal with mandates to have no classes with greater than 40 students?).

  2. Jason Kerwin says:

    >There is a rather large set of important economic problems which don’t have an easy experiment that you could attach to them, but which are still core topics in the field and which economists are still paid to understand. That should be enough to justify the claim that econ is not an experimental science.

    I disagree, and I think that the parachute paper makes this case forcefully. Medicine is certainly an experimental science in that it relies on experiments where it can, but there are hugely important areas of medicine, such as surgery, for which experiments are largely infeasible and rarely used. Economics is certainly further to the right along the experimental-nonexperimental axis, but if anything (true, not quasi-) experiments are underused in the field.

    My main complaint is that he doesn’t actually say what he _means_ by economics not being an experimental science. I think you’re right, and that it sounds like a much stronger claim than he intended.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s