Evaluating Social Impact

(As posted on YourStory.in)

For people engaged in the social entrepreneurship space, one of the most difficult questions is how to measure the positive social impact you make. How do you know you’re doing net social good?

What we typically do is assume that our product has intrinsic positive social value and so simply measure how many people have used our product or service. Then we make grand statements like ‘We have positively touched a million lives’. For some products this might be all it takes. Take d.light, for instance, a solar alternative to kerosene lamps that is cheaper, brighter and healthier. A simple count of product sales would be a pretty good indicator. For many other products and services though it is far more ambiguous. Microfinance, pharmaceuticals, health services, education. All of these have great potential for good but also for abuse, misuse or mistakes. Each instance of use is not always a net positive; a borrower who drinks away a loan, a person who commits suicide with an overdose of painkiler, a person who gets wrong medical advice resulting in a worsening of their condition. In some cases, the net positive impact is highly debated. Are people really better off if they took a loan? If they underwent a particular treatment? Took a particular course?

The Madura Experiment (more refined)

It’s been a bit of a busy month for me and I haven’t had time to post. After years now of reading, surveying, observing and thinking, we (meaning Madura) are finally putting together the pieces of our socioeconomic transformation strategy. We’re taking quite a leap from the run-of-the-mill microfinance, so we’re operating like a start-up again, which is fun. But, with profits to take some chances with, so that’s even more fun. We’re looking at a massive, for profit experiment in reengineering the socioeconomic system dynamics of our members - about half a million poor women in rural Tamil Nadu, moving rapidly to a million. This involves putting in place a large scale smart phone driven data collection system, a mobile phone rollout for our members that will support a host of applications in the future, and development of a cool cutting edge network analysis platform that will allow us to track the evolution of the structure and dynamics of our member network so that we can nudge it towards a more productive trajectory of evolution. So here’s the story, (a bit of a synthesis of many of my previous posts).

Social Impact Evaluation in Interconnected Evolving Systems

Today there are many development economists that focus on social impact evaluation studies. These studies typically compare groups receiving an intervention to groups that did not. For example, did the group that received microfinance do better than the group that did not? Although they attempt to ‘control’ for variability between groups, I see some fundamental flaws in this approach. First, to be meaningful comparison, a ‘control’ group must be independent of the intervention group. In society, which by definition is an interconnected system, this is virtually impossible and you can never control this way for network effects.

For example, lets say the people who received microfinance in the village used the money to buy goods from the people who didn’t. Its not impossible that this could result in the 'control group' getting richer because they could sell more. Compared to them, the microfinance takers could then look poorer because, while they may also have overall bought and sold more, they also had to pay back their loan with interest to the lender. This is a gross and very direct example of course, and it will never be so simple to parse out the network effects. Recognizing this, if you instead decide to move your control group to a distant location this will not really solve the problem. For one thing, the more distant the ‘control’ community, the more the variability in the circumstances and the more confounding this variation can be. But more important, network effects can be far reaching and change dramatically as the size of the network changes. So physical distance does not help unless you compare two communities that are completely self contained economies (in which case they have evolved so separately and so differently that it will be a useless comparison anyway).

This brings me to a second extremely important point. Networks like society are open feedback systems that evolve continuously over time. This means that the impact you find that microfinance has today (network effects notwithstanding) is completely irrelevant to tomorrow and therefore cannot be prescriptive or even diagnostic in any way. For example, cell phone penetration and use is climbing rapidly in rural India today and dramatically changes the way people interact and transact, opening up distant markets and changing the rate at which people can buy and sell. Just imagine the impact of being able to call a buyer in the neighbouring town to coordinate an order and delivery instead of having to travel the three hour distance by bus! So if you conducted your traditional microfinance impact study today, when cell phone penetration was low, maybe it has little impact because the opportunities to interact rapidly were low and therefore business was slow. Tomorrow it may have magnificent impact as the network conditions change.
So, not to sound too harsh, but in the interest of progress, having read various social impact studies on microfinance conducted at great expense and unveiled with great fanfare, I have to say What’s the point? As a practitioner interested in seeding social progress there’s nothing whatsoever I can take from these studies. It doesn’t tell me what is wrong or what should be done right. Rather it is a flawed comparison of one slice in time.

So what’s the alternative? If we want to come up with something valuable prescriptive, what we need to do is stop wasting time with such social impact evaluations and start trying to understand the network and its evolution. What we need to do is begin asking and answering questions like these:
1) What is the topology of the transport and communication networks? How does information and trade flow in these networks? What kinds of simple changes in the network could result in nonlinear benefits for the flow of goods, money and information?
2) How do ideas and innovations diffuse in these populations? How does this compare to more advanced segments of society? What can be done to create more productive diffusion?
3) What do group dynamics look like in these groups? Can we identify aspects of culture and interaction that predict why these groups fail to organize themselves and innovate? If so, what kind of innovations can we develop that shift these behaviours towards more productive dynamics?

Madura is looking for a postdoctoral scientist

Madura is looking for a scientist who will do cool things that haven't been done before. Here's the ad:

Postdoctoral Scientist: Rural Social and Trade Networks

This is a unique position where you will reconstruct the trade and communication networks of rural South India from various large empirical data sets and make it available to the greater scientific community. Real time understanding of the social and trade network topologies and dynamics will be used to design interventions that can significantly impact the dynamics of the network as part of an ambitious large scale social experiment to bring about poverty alleviation.

The key requirements are large imagination, a strong desire to work for social change, the ability to work efficiently with large datasets in MATLAB and C and the ability to design cool GUIs that will facilitate public access to the data.

This position does carry publication potential. However the primary goal is to create understanding that can result in fast productive interventions that positively impact the standard of living.

Please submit a cover letter outlining why this interests you along with your CV to kamalesh@mmfl.in

Spread the word!

A network view of entrepreneurship

Here is a definition of entrepreneurship that I came across in another one of Mark Granovetter's articles called The Impact of Social Structure on Economic Outcomes. He writes:

Schumpeter defined entrepreneurship as the creation of new opportunities by pulling together previously unconnected resources for a new economic purpose.

Granovetter goes on to say:

One reason resources may be unconnected is that they reside in separated networks of individuals or transactions. Thus, the actor who sits astride structural holes in networks (as described in Burt, 1992) is well placed to innovate. The Norwegian anthropologist Fredrik Barth (1967) paid special attention to situations where goods traded against one another only in restricted circuits of exchange. He defined “entrepreneurship” as the ability to derive profit from breaching such previously separated spheres of exchange.

Schumpeter's is an interesting definition and now ranks as the one I like best. Most of the other definitions of entrepreneurship I have seen approach it from the point of view of the outcome (new value created through product innovation etc.) rather than the process of network reconfiguration. The process point of view is more generalizable though because most new businesses are not all that innovative in product and fail many other definitions.

Links back to my post on more connected women being more successful..

The Strength of Weak Ties

I have been thinking a lot about how to understand poverty from the point of view of the properties of the social network. In this context, I thought I would share with you a very important paper by sociologist Mark Granovetter written in 1973 called ‘The Strength of Weak Ties’ which he has more recently revisited in a new paper called ‘The Strength of Weak Ties: A Network Theory Revisited’. Here is an excerpt:

.....individuals with few weak ties will be deprived of information from distant parts of the social system and will be confined to the provincial news and views of their close friends. This deprivation will not only insulate them from the latest ideas and fashions but may put them in a disadvantaged position in the labor market, where advancement can depend, as I have documented elsewhere (1974), on knowing about appropriate job openings at just the right time. Furthermore, such individuals may be difficult to organize or integrate into political movements of any kind, since membership in movements or goal-oriented organizations typically results from being recruited by friends. While members of one or two cliques may be efficiently recruited, the problem is that, without weak ties, any momentum generated in this way does not spread beyond the clique. As a result, most of the population will be untouched. The macroscopic side of this communications argument is that social systems lacking in weak ties will be fragmented and incoherent. New ideas will spread slowly, scientific endeavors will be handicapped, and subgroups separated by race, ethnicity, geography, or other characteristics will have difficulty reaching a modus Vivendi.

I share this because villages, which are typically poor by nature, are generally insular, tending to rely much more on strong ties with very few weak ties outside their village. Given this, one strategy Madura takes as an organization is to bring in products, services and events that foster ties across villages and to the urban economy. This may all sound like common sense but what is surprising, if you get into the models and workings of networks, is just how profound the consequences of a few weak ties can be. (I've spent the last five years thinking about this mostly in the context of the brain and it is interesting to note that the cerebral cortex, the outer layer of the brain where all the high level thinking gets done, is characterized largely by weak connections).

Connected we succeed, divided we default

At Madura Microfinance, one of our primary assumptions is that women who are more informed and better connected will be more successful and make more productive use of loans. So, much of our efforts are aimed at increasing networks and access to information among our members. A PhD Student from Oxford, Sangamitra Ramachander, recently studied our women’s borrower groups to see what kind of factors predicted whether a group went successfully on to the next higher level loan or would default. This is still a work in progress but there are some very interesting results. Here is one odd one that stood out to me. She found that women that travelled more frequently to neighbouring villages (but surprisingly not the nearest towns) were several times more likely to be successful rather than default. It’s not clear whether this factor is causal or the outcome of their success but it’s something worth exploring further. Possibly, women who travel more to neighbouring villages are more informed about local markets and better connected within them?

The Physics of Friendship: Do we make friendships no differently than a bunch of particles bouncing around?

Here's an interesting article that I came across that surveyed 90,000 high school students to reconstruct friendship networks and then created a model of the results using knowledge of how particles in physical systems collide and interact. Haven't read the article in depth so I'm not really doing much of a job in terms of useful commentary on its merits. Just pointing it out. (The lead author on the paper though is Marta Gonzales who worked with Albert Laszlo Barabasi who wrote Linked that I posted on earlier).

The Physics of Friendship (This is an easy to read writeup on the actual paper)
By comparing people to mobile particles randomly bouncing off each other, scientists have developed a new model for social networks. The model fits with empirical data to naturally reproduce the community structure, clustering and evolution of general acquaintances and even sexual contacts.

The paper is also an example of how you can use surveys to reconstruct networks and dynamics, relevant to my previous post 'Where does the money go?' and the discussion around it. It would be interesting to see how such social networks in small villages differ from those in large cities particularly with respect to trade (as opposed to friendships). Could be quite telling about how our microentrepreneurs do business since how they bump into each other would be totally different from the way we technology enabled urban folk do.