Evaluating Social Impact

(As posted on YourStory.in)

For people engaged in the social entrepreneurship space, one of the most difficult questions is how to measure the positive social impact you make. How do you know you’re doing net social good?

What we typically do is assume that our product has intrinsic positive social value and so simply measure how many people have used our product or service. Then we make grand statements like ‘We have positively touched a million lives’. For some products this might be all it takes. Take d.light, for instance, a solar alternative to kerosene lamps that is cheaper, brighter and healthier. A simple count of product sales would be a pretty good indicator. For many other products and services though it is far more ambiguous. Microfinance, pharmaceuticals, health services, education. All of these have great potential for good but also for abuse, misuse or mistakes. Each instance of use is not always a net positive; a borrower who drinks away a loan, a person who commits suicide with an overdose of painkiler, a person who gets wrong medical advice resulting in a worsening of their condition. In some cases, the net positive impact is highly debated. Are people really better off if they took a loan? If they underwent a particular treatment? Took a particular course?

Let’s take microfinance as an example, for two reasons: first, I know it well and second, it’s attracted a great deal of attention from academics, mainly developmental economists, interested in evaluating its impact. Ideally you want a social impact measure that can be diagnostic (am I having net positive impact?) and also prescriptive (what should I do differently if I’m not?). Economists will typically evaluate whether a group that received microfinance (the intervention group) did better than the group that did not (the control group). As a scientist I have a list of quibbles with the methodologies and value of such studies.

First, to be meaningful comparison, a ‘control’ group must be independent of the intervention group. In society, which by definition is an interconnected system, this is virtually impossible. For example, let’s say the people who received microfinance in the village used the money to buy goods from the people who didn’t. It’s not impossible that this could result in the 'control group' getting richer because they could sell more. Compared to them, the microfinance takers could then look poorer because, while they may also have had some business benefit, they had to pay back their loan with interest to the lender too. Recognizing these ‘network’ effects, if you instead decide to move your control group to a distant location, this raises another set of issues – the more distant you go, the more differences in the way the people will behave and therefore make use of their money.

Second, society is not static but evolves over time. This means that the impact you find that microfinance has today (network effects notwithstanding) is completely irrelevant to tomorrow and therefore cannot be prescriptive. For example, if you did the study in a village before the bus service began, borrowers would have had very little opportunity to access larger markets and the loan would have been unproductive. After the bus, it could all change.

So as a microfinance company interested in understanding social impact in a diagnostic and prescriptive way, what are we to do? My view is that instead of focusing on the net outcome of product with pointless retrospective social impact evaluations, we should focus on understanding the drivers of positive use of the product. What kind of village ecosystem is best suited for microfinance? What profile of student or community does best with this kind of education? What kind of patient profile will respond best to this treatment? It’s these kinds of insights that are prospective and help us get better.