Social Impact Evaluation in Interconnected Evolving Systems

Today there are many development economists that focus on social impact evaluation studies. These studies typically compare groups receiving an intervention to groups that did not. For example, did the group that received microfinance do better than the group that did not? Although they attempt to ‘control’ for variability between groups, I see some fundamental flaws in this approach. First, to be meaningful comparison, a ‘control’ group must be independent of the intervention group. In society, which by definition is an interconnected system, this is virtually impossible and you can never control this way for network effects.

For example, lets say the people who received microfinance in the village used the money to buy goods from the people who didn’t. Its not impossible that this could result in the 'control group' getting richer because they could sell more. Compared to them, the microfinance takers could then look poorer because, while they may also have overall bought and sold more, they also had to pay back their loan with interest to the lender. This is a gross and very direct example of course, and it will never be so simple to parse out the network effects. Recognizing this, if you instead decide to move your control group to a distant location this will not really solve the problem. For one thing, the more distant the ‘control’ community, the more the variability in the circumstances and the more confounding this variation can be. But more important, network effects can be far reaching and change dramatically as the size of the network changes. So physical distance does not help unless you compare two communities that are completely self contained economies (in which case they have evolved so separately and so differently that it will be a useless comparison anyway).

This brings me to a second extremely important point. Networks like society are open feedback systems that evolve continuously over time. This means that the impact you find that microfinance has today (network effects notwithstanding) is completely irrelevant to tomorrow and therefore cannot be prescriptive or even diagnostic in any way. For example, cell phone penetration and use is climbing rapidly in rural India today and dramatically changes the way people interact and transact, opening up distant markets and changing the rate at which people can buy and sell. Just imagine the impact of being able to call a buyer in the neighbouring town to coordinate an order and delivery instead of having to travel the three hour distance by bus! So if you conducted your traditional microfinance impact study today, when cell phone penetration was low, maybe it has little impact because the opportunities to interact rapidly were low and therefore business was slow. Tomorrow it may have magnificent impact as the network conditions change.
So, not to sound too harsh, but in the interest of progress, having read various social impact studies on microfinance conducted at great expense and unveiled with great fanfare, I have to say What’s the point? As a practitioner interested in seeding social progress there’s nothing whatsoever I can take from these studies. It doesn’t tell me what is wrong or what should be done right. Rather it is a flawed comparison of one slice in time.

So what’s the alternative? If we want to come up with something valuable prescriptive, what we need to do is stop wasting time with such social impact evaluations and start trying to understand the network and its evolution. What we need to do is begin asking and answering questions like these:
1) What is the topology of the transport and communication networks? How does information and trade flow in these networks? What kinds of simple changes in the network could result in nonlinear benefits for the flow of goods, money and information?
2) How do ideas and innovations diffuse in these populations? How does this compare to more advanced segments of society? What can be done to create more productive diffusion?
3) What do group dynamics look like in these groups? Can we identify aspects of culture and interaction that predict why these groups fail to organize themselves and innovate? If so, what kind of innovations can we develop that shift these behaviours towards more productive dynamics?