Thoughts on GIIN's "Impact Measurement (IM) and Performance Management Training"

At Upaya, the process of collecting and assessing beneficiary data to track outcomes and measure impact is fundamental to our work. After all, four years ago the organization was founded with a hypothesis that providing jobs -- and not handouts -- is the most efficient and effective way for the “ultra poor” to progress out of poverty. To date, we have sought to prove or disprove this hypothesis through social performance measurement (what we internally refer to as “SPM”), the systematic collection and analysis of beneficiary level information.

I recently had the wonderful opportunity of attending the “Impact Measurement (IM) and Performance Management Training” in Bangalore organized by the Global Impact Investing Network (GIIN) in collaboration with international consultancy Steward Redqueen and Social Value International, to reflect on Upaya’s methodology and revisit some of these ideas. The purpose of the training was to introduce practitioners to different IM frameworks and ways in which these frameworks could be used to suit the context of an individual organization and its mission.

Some of my key takeaways from the training were:

• Don’t overstress the rigour; it should be “good enough”
• Data is not just for reporting but also for decision making
• Consider establishing a counterfactual in the setting

Financial and business decisions are made everyday on the basis of imperfect information provided by financial accounting frameworks. Yet when it comes to evaluating impact, we tend to hold ourselves to a higher standard or rigour --where randomized control trials are considered to be the gold standard. However, sometimes the fear of publishing imperfect impact data gets in the way of doing any IM activity. If the data is ”good enough” to spot trends, aid in decision-making and undertake some course correction, and not overly tax human or financial resources, then it is worth undertaking.

Secondly, these days it appears that impact data is mostly used for reporting outwardly to stakeholders, such as philanthropists and investors. In doing this, we overlook the critical role data can play in helping an entrepreneur and/or management team make decisions about the business itself. The challenge most often cited among practitioners is one of constrained resources - the buy-in needed to undertake the exercise of data collection and the need for doing it all. For an early stage enterprise that is working to expand the business, build the team, formulate marketing strategy, and raise funding, IM is a weighty commitment.

Data collected through impact measurement, however, should not be seen separate and distinct from business data. Instead, if an IM framework were designed to provide useful insights for the business about its beneficiaries, that could then inform refinements to the core product or service, then there exists a higher probability of entrepreneur’s buy-in and also better quality data.

Last, but certainly not least, is the need to accommodate the counterfactual in the actual impact assessment. A counterfactual simply put means “what would have happened anyways, without the intervention in question.”  A counterfactual could be in the form of state and national averages based on government data. Or it could be a more statistically rigorous “control group” or “non- intervention group.” The counterfactual, when compared with data that depicts the outcomes of the intervention, can give us a clearer indication of impact, or what positive effects can be attributed to the intervention. In essence, it can provide the necessary context and help us paint a picture of the impact that has occurred.

In the coming months, we will integrate some of these learnings into our SPM framework. We hope to make our system of impact assessment more robust so as to provide high quality information on the progress made out of poverty by our beneficiaries. Our goal is to simultaneously provide valuable insights and learnings to our entrepreneurs, to support the continued growth of their businesses and the creation of hundreds more jobs.

"Tamul Plates Social Impact Report: 2014 Baseline" Now Available

Right Click > "Save As" to download the full report

During the summer of 2014, the Upaya staff and Tamul Plates management began work on this Social Performance report to document the economic and social background of the company’s beneficiaries. This report provides a snapshot of social metrics for 95 of Tamul Plates’ beneficiaries, serving as the baseline for reporting their progress out of poverty over time. 

Surveyors interviewed Tamul Plates beneficiaries across ten districts of Northeast India. For the purposes of this report, the respondents were broken into two geographic groups - the Lower Assam Group and the Upper Assam Group. Beneficiaries were evaluated across key social and economic metrics, including income, education, assets, and expenditure. Among the highlights:

• Households in the Lower Assam Group have a higher likelihood of falling below the $1.25-a-day poverty line than those in the Upper Assam Group. In particular, leaf plate producers (32 percent in the Lower Assam Group, and 36 percent in the Upper Assam Group) and raw material collectors (28 percent in both respondent groups) are the most likely to fall below the poverty line among Tamul Plates beneficiaries.
• Leaf plate producers are highly dependent on Tamul Plates, with income from Tamul Plates-related activities constituting the primary source of income for a vast majority of households across the Lower and Upper Assam Groups.
• Households spend roughly 50 percent of their total monthly expenses on food alone. Expenditure on school for children and miscellaneous (unplanned) expenses form the next two biggest categories. Savings constitute a very small component of total expense for households at 6–7 percent. 

A midline (check-in) survey will be conducted after 12 months for the same group of beneficiaries to measure the changes in income levels and quality-of-life indicators. Click here to download the full report.

What We're Reading August 2014: Summer Breeze

Stanford Graduate School of Business “Does Impact Investing Really Have Impact?” (4 August 2014)

Jyotsna came across this YouTube video of a panel discussion on Impact Investing from the 2013 Social Innovation Summit at the Stanford Graduate School of Business. It led to a good debate around what the merit of impact investment really is, and whether or not it could become a mainstream asset class or another form of venture philanthropy. 

Jyotsna also took pride in noting that Michael Smith from the White House Social Innovation Fund is exhorting exactly the type of evidence-based practice that Upaya holds central to its approach through continually measuring outcomes, refining models, and improving the business program to maximize benefit. He also warns of a trend he sees in investors who seek out innovation by “running after bright shiny objects and creating things people don’t need,” and instead is promoting efforts that are focused on measurably better, outsized positive effects for the public. We couldn’t agree more with this approach.

The Nand & Jeet Khemka Forum Podcast “Artha Initiative: Investing For Impact with Audrey Selian” (August 2014)

Sreejith sent around this great audio interview with Audrey Selian, Director of Rianta Capital’s Artha Challenge in which Audrey hits on some really interesting insights about impact investing. In particular, we felt her comments on sacrificing good investments in search of perfect investments were spot on. She also made some nice points on creating social value in areas where most infrastructure and services are non-existent. Definitely worth the read.

Stanford Social Innovation Review “Fundraising is Fundamental” (26 February 2014)

Laurel sent this piece around to the team in early August. She was struck by the way the authors touted the interesting correlation between forward thinking, innovative nonprofits and discomfort around fundraising.

“The organizations that have the most compelling logic models and the most impressive record of impact (as demonstrated by external impact evaluations) tend to be the worst at raising money—and vice versa ... At many bold and extraordinary nonprofits, people cease to be bold when the topic of fundraising comes up.”

The authors throw out a number of ideas for overcoming this "unfortunate inverse correlation." One strategy is approaching external actors as potential collaborators and passionate partners in the fight to end poverty, instead of as potential funders. This attitude can humanize all players and lead to deeper personal relationships. It also opens up an organization to support in a wider variety of forms - be it moral, in-kind, strategic or material. Of course, she noted that we at Upaya also accept snacks.

New Frontiers: Measuring Progress Beyond the Pay Packet

Authored by Upaya's own Jyotsna Taparia, this article was submitted in June to the Devex/ USAID "Frontiers in Development" Essay Competition.  The competition prompted writes to submit their thoughts on a variety of questions under the heading How would you Eradicate Extreme Poverty by 2030? 

Q: Aside from income, how might we define and measure other dimensions of extreme poverty?

“Development requires the removal of major sources of unfreedom: poverty as well as tyranny, poor economic opportunities as well as systematic social deprivation, neglect of public facilities as well as intolerance or overactivity of repressive states.”

-Amartya Sen (1999), Development as Freedom

For far too long the discourse on poverty has been limited to income or lack of it thereof. The discourse on extreme poverty or absolute poverty has been taking its shape and form since early 80’s. In 1990, the World Bank proposed that global poverty should be measured through the standards of the poorest countries and arrived at a $1 a day poverty threshold, a figure that was last updated to $1.25 a day (based on 2005 PPP)[1]. This definition also became the basis for Millennium Development Goal #1: reduce by half the proportion of extreme poor (those living under $1.25 a day) by 2015.

However, income measures can only go so far as to capture the consumption capacity of an individual, calculated either in monetary terms or nutritional count. They are grossly insufficient in capturing extreme poverty as they do not exhibit any sensitivity towards the depth[2], duration and direction[3] of poverty.

In his seminal work Development as Freedom (1999), Nobel laureate Amartya Sen outlines how the development debate should be structured. Sen postulates that development is closely linked to three sets of freedom: economic, social and political. Poverty in this framework is described as absence of at least one freedom. According to Joseph Wrensinski, a lifelong activist and founder of the ATD Fourth World, extreme poverty is a “... lack of basic security [that] simultaneously affects several aspects of people’s lives, when it is prolonged and when it severely compromises people’s chances of regaining their rights and of reassuming their responsibilities in the foreseeable future.” The underpinnings of this approach are largely similar to what Sen proposes - poverty is deeper than just the state of material deprivation and is not static in time.

From the postulations of Sen and Wrensinski, it’s clear that extreme poverty is a result of three crucial factors:

a. Availability and efficiency of human, financial and physical assets

b. Inequality in the availability of opportunities and ability to exercise agency

c. Interaction with measurable deprivations that reinforce the impact of others

Despite the longstanding focus on income as the sole indication, a significant body of work has emerged establishing the multidimensional nature of poverty both at the household and at the community level. However, identification of the extremely poor based on this multidimensionality poses its own set of unique challenges. For example, if the indicators being used are income, school attendance, nutritional status and health status, then there are some scholars who argue that a household falling below the minimum threshold on any one of the indicators should be considered poor. There are still others who contend that households should score low on all indicators in order to qualify as poor. With these conflicting approaches to poverty identification, one runs the risk of erroneously including or excluding a fraction of the poor population when developing programme interventions (also known as an error of commission or omission.)

Extreme poverty measurement is much more complex than a simple error of omission or commission. It has been observed that deprivation of one indicator actually has negative impact on other indicators, resulting in a self-perpetuating cycle of poverty that is often referred to as the “vicious cycle of poverty.” Thus any discussion of alternative measures must look at these trailing indicators as well as leading ones. These additional indicators can not only capture the effect of extreme poverty, but also show us the progress being made by a household. For example, a household that cannot afford to send its children to school will see them working either with their parents or elsewhere. The lack of formal education and skills won’t allow them to compete in the more remunerative skilled job market and will often result in a lower household income.

Following this logic, it is useful to look deeper at some of the trailing indicators of extreme poverty that are common to all contexts and benchmark them against established trends, such as:

●      Households that typically spend more than 50% of their income on food expenditure are more sensitive to income shocks and less likely to avail of services like health and education. Deprivation for these households would be on multiple counts - lack of food, education, quality health care and other services.  Therefore an increase in income should result in a decline in the food expenditure to income ratio[4] but also a concurrent increase in the uptake of the other services.

●      Households relying on manual labour (informal and unorganized) as their primary source of income are more likely to be in the extreme poor category as the availability of work is not only infrequent and erratic in nature but also low paying. Therefore, tracking changes in the nature of the work that generates income for the family can provide valuable insights.

●      The presence or absence of certain classes of assets is also an indicator of the extent of poverty in the household. A low percentage ownership of productive assets (land, livestock, simple machinery and tools etc.) is a likely trailing indicator of extreme poverty. 

●      Households rate of electrification against local and regional statistics. Electrification has a direct impact on trailing indicators - household assets, cooking and refrigeration, educational success - and therefore is often prioritized by households as income stabilizes or increases. Admittedly, the legality of such connections is often murky at best, but it is nonetheless an indicator of a household’s day-to-day income situation.

Because extreme poverty is relative, we must look at each case in the context of a larger community. Geography and surroundings play an important role in determining the common minimum threshold for a poverty measure or even if the measure will prove to be valuable in providing insights into the extent of deprivation. Therefore, the goal of poverty measurement should not be to create a one-size-fits-all multidimensional index but rather a set of robust indicators that is most relevant to the local context. While this route may not allow for seamless cross comparison, it is successful in achieving a high degree of universality.

---

[1]Ravillion et. al (2008). Dollar a Day revisited, http://www-wds.worldbank.org/servlet/WDSContentServer/WDSP/IB/2008/09/02/000158349_20080902095754/Rendered/PDF/wps4620.pdf

[2]Depth of poverty is related to the extent by which a household falls below the poverty line threshold.

[3]Households also show movement out of poverty to fall back again due to external shocks (for example, detection of an ailment with prolonged treatment, natural calamities etc.)

[4]For a richer discussion on this refer to http://www.ers.usda.gov/publications/err-economic-research-report/err89.aspx

Editorial for moneyspentwell.org: Strengthening the silk industry and its workers

This article was originally published 7 April 2014 on moneyspentwell.org.

The sleepy town of Bhagalpur, India, is famous for two things – fertile lands and fine silks. The second largest city in Bihar, Bhagalpur has earned the title of ‘Silk City’ for the high quality of Tussar Silk – a high-luster, strong, lightweight copper silk that wears very well in tropical heat, most valuable when woven by hand. Weaving in Bhagalpur is an art that has been passed on for generations. Working on large wooden pit-style looms found across the city that have been with families for decades, parents share the secrets of crafting fine Tussar Silk scarves and sarees. In fact, there are an estimated 30,000 handloom weavers and about 25,000 handlooms in Bhagalpur.

A silk weaver at his loom in Bhagalpur, Bihar, India. Courtesy: Upaya SV.

However, Tussar Silk weaving is also a dying trade in Bhagalpur as the majority of those 25,000 looms sit idle, unable to provide a livelihood for their owners. It is estimated that that market for sarees in the country stands at $2 billion and poised to grow, but the move to less delicate power looms and an influx of cheap imported chinese silks have flooded the domestic saree market.

Even in the face of this competition, a niche market that values handcrafted products remains. However, exploitative supply chains and a lack of market linkages to wholesale buyers have made it impossible for weavers to earn a viable and dependable living from their work. As such, the average weaver earns less than 25% of the final sale price of a saree that takes weeks to create. Furthermore, without continual skill-building and access to new materials, there is no way for the weavers to build skills needed to meet changing consumer trends and preferences.

Bhagalpur itself has seen very little industrial development, resulting in widespread migration to other parts of the country. As a result many weavers have left the trade altogether, migrating to urban centers to find construction work or taking up farming far from Bhagalpur. In fact, in 2004 the Government of India named Bhagalpur in the list of country’s 250 most backward (note: poor) districts (of a total of 640).

Those who continue weaving do so part time, cobbling together manual labor jobs and other unskilled activities to earn a living. One recent survey has indicated that most weaving households live on less than Rs. 3000 ($50) each month.

Compounding this situation is the fact that payments received from middlemen – local traders intent on buying products as cheaply as possible, regardless of their quality – are opaque and erratic.  Many weavers complain that, not only do they not receive payment that they deserve, and are forced to make multiple visits to these middlemen to get their due. Facing the prospect of being seen as “troublesome” and losing their current source of livelihood, many do not pursue the matter and allow the cycle of exploitation to continue.

Eco Kargha weaver prepares bobbins of silk thread in Bhagalpur, Bihar, India. Courtesy: Upaya SV

In November 2012, Upaya Social Ventures initiated a partnership with Bhagalpur based start-up Eco Kargha Marketing Private Limited headed by Dr. Ravi Chandra, a passionate Bihar native with a strong desire to see his state thrive. Hailing from the capital city of Patna, Ravi has worked tirelessly to create institutions in Bihar that can effect large scale economic growth and development in the state.

Eco Kargha was set up to improve the quality of life for rural weavers by providing the linkages and resources for the modernization of the ailing traditional handloom industry. The company trains marginalized Tussar Silk weavers on new skills, techniques, equipment and designs for producing high quality products for the modern retail marketplace. Eco Kargha also manages relationships with large national retailers such as Fab India and ANS Exports, bringing in bulk orders and ensuring that weavers can earn a full-time living from their work at the loom.

“Customers across India know the quality of Tussar Silks and are ready to pay handsomely for them. Weavers in Bhagalpur are extremely talented and ready to produce the garments. All we are doing is bringing those producers and consumers together in a beneficial way,” said Dr. Chandra.

The company is able to break the stranglehold that middlemen and traders have on the industry by directly working with the artisans. By forming formal weaver groups with a master weaver at helm, these groups in collective are ensured of a steady stream of work and greater bargaining power. Through normalizing payments and improving transparency, Eco Kargha is tilting the economics of weaving back in the favor of the weavers. It has also worked to provide additional services – opening bank accounts, obtaining medical insurance cards – to these weavers through partnerships with existing Central and State Government programs.

In one year of operations it has been successful in earning revenues of over Rs. 80 lakhs ($130,000) by selling fabric, sarees and scarves to large export houses and established retail chains throughout India. To increase its footprint Eco Kargha has spent considerable time and effort on a conscious blend of B2B and B2C sales. The company has also launched Eco Stree, its in house brand of saree and scarves. Through their work they have been able to provide steady and predictable source of employment for over 100 weavers and increase their income levels by almost 50%.

One of the biggest pain points for Eco Kargha and other businesses that work directly with weavers is the high requirement for working capital.  A big component of the raw materials costs is the cost of yarn and dyes. This upfront advance payment constitutes almost 50% of the value of the order processed, but puts a strain on the company’s cash reserves. Due to the early stage of the company, financial institutions and lenders are reluctant to extend a line of credit.

Eco Kargha also faces the challenge of breaking into a saree market that has been dominated by a handful big players for decades. India’s saree market is estimated to generate $2 billion per year in sales and is projected to grow at 8.5% per annum. Within that market, Bhagalpur-made Tussar Silks remain a specialty product despite their nationwide renown. To tap this behemoth industry with deep-rooted interests is a challenge. To overcome the obstacles, Eco Kargha is assembling a professional and dynamic sales team with the right blend of industry experience and fresh talent.

Eco Kargha is projecting a growth rate of over 100% over the next two years, however, it will require equity investment up to $200,000 and an additional $160,000 in working capital debt to meet its goals.  The investment will allow Eco Kargha not only to ramp up its production capacity, but also to build infrastructure like dying units and establish a retail brand presence that will further enhance its competitive edge. Best of all, if Eco Kargha’s growth continues as projected, the company will be able to provide dignified employment to over 500 weavers and provide their families with a real path out of extreme poverty.

Social Performance Metrics report 24% increase in the per capita income of MILK ROUTE dairy farmers

Committed to rural transformation, MokshaYug Access (MYA), the company behind the brand MILK ROUTE, in association with Upaya Social Ventures, released its first Social Performance Metrics report for 2012-2013. MYA undertook this evaluation to measure the social impact of its dairy business and assess the benefits MILK ROUTE has had on the lives of dairy farmers. The survey was conducted over a course of four months in 2012 and covered a total of 1,486 households across 269 villages in the districts of Tumkur, Mandya and Kolar.

According to key findings of the survey:

• MILK ROUTE households reported a 24% higher average per capita income when compared to their counterparts (Control Group households).
• By virtue of access to high quality cattle feed and veterinary services, MILK ROUTE households produced milk that consistently beat the industry’s average yield parameters, as measured by percentage fat and non-fat solids. These higher yields provided an additional 10% in earnings per litre for the households.
• MILK ROUTE households were also able to provide, on average, 5.5 more litres of milk per month to their children compared to households that were not suppliers of the MILK ROUTE dairy. The choice to consume more at home versus selling the milk demonstrated greater income security in general.
• As a result of increased, regular and transparent compensation combined with a support system that included access to training, high quality feed, and veterinary services— nearly 41% of all MILK ROUTE households reported dairy as their primary livelihood versus 25% of households before MILK ROUTE was introduced.

Harsha Moily, Founder and CEO, MokshaYug Access said, “One of MYA’s key objectives is to create income certainty for the farmers through our First-mile focus. It is encouraging to know that our efforts of transforming rural households are paying off. The social performance metrics survey helped us analyze our progress in our farmer centric approach and also helped us understand the gaps and what we need to do next to take this to the next level.”

Sachi Shenoy, Executive Director, Upaya Social Ventures added, “We believe in partnering with businesses that transform lives by creating dignified job opportunities. It is encouraging to see that MYA has had an impact on the lives of so many dairy farmers in Karnataka. This is just the beginning and we look forward to growing our association with MYA.”

METHODOLOGY: The data collection for MILK ROUTE involved both quantitative and qualitative approaches and relied on the usage of standard survey tools to directly collect and assimilate data from farmers. The SPM evaluation was carried out in three districts of Karnataka, namely Kolar, Mandya, and Tumkur. A total of 1,486 households were surveyed from 269 villages and fall into one of two groups: those who are suppliers of the MILK ROUTE dairy and those who are not (Control Group).

The process of narrowing down which MILK ROUTE households to survey included three tiers of selection criteria. In the first tier, milk procurement estimates were taken for each district to help in the selection of MILK ROUTE villages. Based on the percentage of milk procured from each district, a proportional number of sample villages were selected. Thus, 49% of the MILK ROUTE households were selected from Tumkur, 40% from Kolar, and 11% from Mandya, which roughly aligns with the fact that 48% of the total milk collected by MILK ROUTE comes from Tumkur, 44% from Kolar, and 8% from Mandya. In the second tier, the selection of individual villages was further qualified through purposive sampling. The only villages eligible for selection were ones that provided over 80 litres of milk per day to MYA. The third tier involved a random selection of households based on the short list of villages prepared through purposive sampling.

Control Group villages were selected in order to establish a proxy baseline to aid and assist in comparison. The selection of Control Group villages was completed through a matching process. This exercise involved each MILK ROUTE village being matched to a Control Group village based on key parameters such as economic activity, demographic profile, access to infrastructure, and distance to the nearest city. The selection of Control Group households was also purposive, i.e. those who did not own cattle were screened out.

To download a copy of the report,

click here

.