It’s been exciting watching the economics world embrace randomized controlled trials (RCTs), especially in evaluating practices in the developing world. But is economics (and more broadly, behavioral science) suited for evaluating large-scale questions through RCTs?
One thing that has always appealed to me about academic medicine is the high level of rigor we apply toward evidence evaluation. While sometimes RCTs are impossible to conduct (anyone want to be randomized to be in the smoking arm of a trial to see if tobacco increases mortality?), decision makers tend to rely heavily on RCTs when setting guidelines.
But what do the results tell us? If I really want to compare two groups, I want almost everything to be the same between them- except for the intervention we’re testing. If one group has too many men, too many senior citizens or too many diabetics compared to the other group, how do we know any observed difference is attributable to the intervention? One way to achieve homogeneity is by having strict inclusion criteria (e.g., only women between the ages of 40 and 60) to keep the groups similar, but this is fraught with issues. The more we control our controlled trials, the less generalizable our results are. If a study only included women, how would we know if the results apply to men?
Because global health and development research is often widely applied (a study done in Uganda may be used to justify a policy change in Namibia), generalizability becomes paramount. It would be naive to think medicine is immune to these issues. For example, approximately 20% of Botswana citizens have a genetic makeup that causes them to more slowly metabolize efavirenz, a key drug in southern Africa’s HIV armamentarium. A study assessing efavirenz’s efficacy may yield very different results in Botswana than in neighboring Zimbabwe.
This issue of heterogeneity and lack of generalizability, however, may be more significant in economics research. In essence, economics is the study of utility and efficiency. These are culturally laden issues and the questions economists ask are equally culturally laden. Because norms and mores vary so greatly throughout the world, a result obtained in one setting may have no bearing on another setting.
For a time it was fashionable to suggest that charging a small amount for condoms or mosquito nets or other items might increase usage, because it would give the items more perceived value (whereas free items were thought to be viewed as valueless). Jason wrote about this and showed some economics studies to the contrary. I liked the post but had questions. With only a handful of studies, the majority from Kenya (most examining different products), what conclusions can we draw? Maybe people in Kenya didn’t need things to cost money to convince them to use them, but could the same be said about neighboring Tanzania? What about Zambia? Surely, different countries (and diverse groups within those countries) have varying views on cost and value.
The conclusions here may not be profound. Sometimes a study only tells us so much, and we should use caution before applying the results too broadly. But I think it’s worth questioning if asking behavioral rather than medical questions invites greater heterogeneity .
There’s a fine counterargument here, namely that maybe we shouldn’t focus so much on generalizability. Maybe we should make our studies more granular; instead of trying to answer a question about Kenya (or any other country), we should try to focus on one ethnic group in one geographic area of the country. The results from such a study may not be applicable to the rest of the world, but we could be relatively confident of what to do with that one population. But with limited funds and so many worthwhile questions to answer, the appeal of generalizability is obvious. Is the field of economics up for the task? Is the field of medicine?