Social Impact Evaluation in Interconnected Evolving Systems

Today there are many development economists that focus on social impact evaluation studies. These studies typically compare groups receiving an intervention to groups that did not. For example, did the group that received microfinance do better than the group that did not? Although they attempt to ‘control’ for variability between groups, I see some fundamental flaws in this approach. First, to be meaningful comparison, a ‘control’ group must be independent of the intervention group. In society, which by definition is an interconnected system, this is virtually impossible and you can never control this way for network effects.

For example, lets say the people who received microfinance in the village used the money to buy goods from the people who didn’t. Its not impossible that this could result in the 'control group' getting richer because they could sell more. Compared to them, the microfinance takers could then look poorer because, while they may also have overall bought and sold more, they also had to pay back their loan with interest to the lender. This is a gross and very direct example of course, and it will never be so simple to parse out the network effects. Recognizing this, if you instead decide to move your control group to a distant location this will not really solve the problem. For one thing, the more distant the ‘control’ community, the more the variability in the circumstances and the more confounding this variation can be. But more important, network effects can be far reaching and change dramatically as the size of the network changes. So physical distance does not help unless you compare two communities that are completely self contained economies (in which case they have evolved so separately and so differently that it will be a useless comparison anyway).

This brings me to a second extremely important point. Networks like society are open feedback systems that evolve continuously over time. This means that the impact you find that microfinance has today (network effects notwithstanding) is completely irrelevant to tomorrow and therefore cannot be prescriptive or even diagnostic in any way. For example, cell phone penetration and use is climbing rapidly in rural India today and dramatically changes the way people interact and transact, opening up distant markets and changing the rate at which people can buy and sell. Just imagine the impact of being able to call a buyer in the neighbouring town to coordinate an order and delivery instead of having to travel the three hour distance by bus! So if you conducted your traditional microfinance impact study today, when cell phone penetration was low, maybe it has little impact because the opportunities to interact rapidly were low and therefore business was slow. Tomorrow it may have magnificent impact as the network conditions change.
So, not to sound too harsh, but in the interest of progress, having read various social impact studies on microfinance conducted at great expense and unveiled with great fanfare, I have to say What’s the point? As a practitioner interested in seeding social progress there’s nothing whatsoever I can take from these studies. It doesn’t tell me what is wrong or what should be done right. Rather it is a flawed comparison of one slice in time.

So what’s the alternative? If we want to come up with something valuable prescriptive, what we need to do is stop wasting time with such social impact evaluations and start trying to understand the network and its evolution. What we need to do is begin asking and answering questions like these:
1) What is the topology of the transport and communication networks? How does information and trade flow in these networks? What kinds of simple changes in the network could result in nonlinear benefits for the flow of goods, money and information?
2) How do ideas and innovations diffuse in these populations? How does this compare to more advanced segments of society? What can be done to create more productive diffusion?
3) What do group dynamics look like in these groups? Can we identify aspects of culture and interaction that predict why these groups fail to organize themselves and innovate? If so, what kind of innovations can we develop that shift these behaviours towards more productive dynamics?

The Value of Money

Yesterday my mother called me to say that she is taking my six year old son to see a coin collector since he has been pestering her with questions about money. Motivated by his interest in buying yet another Ben 10 toy that I won’t get for him (jet ray, shake ray, I don’t get it...), he has become a little obsessed with finding ways to get money. Who invented it? Can he invent his own money, he wants to know. Would people take his invented money? Who makes the money? Can’t he just print some out on our printer? Who decides what its worth?

So back to the coin collector. My first reaction was that it seems a little silly and boring. What’s he going to show him, some bunch of coins from around the world? So what? But then I took a look at some of the coins and turns out it’s a very fascinating lesson in the history of money that sparked some interesting breakfast conversation today. The coins are all from ancient civilizations around the world and hark back to the first invention of money. Here are some of them:



They’re not perfect in shape and are pretty crude. You could so easily counterfeit these, was my first reaction. But that’s OK, my husband said, these coins have intrinsic value. They’re worth the metal they’re made from – x grams of brass or silver for instance. So if you wanted to counterfeit them, you would have to spend the equivalent value of energy and time to mine the metal and that has worth exactly equal to its value.

Our typical text book definition of money is as a store of value. In the last few years I have been thinking of money as being a proxy for potential energy that we accumulate on some level – since it provides the power to do work and reconfigure things. So that money should have intrinsic value (like metal) makes sense to me. You could actually calculate its value in units of energy, say joules. Today we have decoupled currency from any ‘actual value’ and created fiat money. Which means money is no longer linked to something with intrinsic value like gold bars. Given this, its ‘value’ can get progressively more distorted. With this disconnect it takes on a different life and dynamics that I’m still trying to get my head around.

So to my son, I have to say, ‘I have some notional answers to your questions and this is how I’m thinking about it, but in all honesty, sweetheart, I’m almost as baffled as you are. Why don’t you go ask Dad.’

'And, hey, I already told you that we are not going to buy another Ben 10 toy right now'.