UPAYA FUNDAMENTALS

In this blog series, members of the Upaya team dive into the what, why, and how of Upaya’s mission to lift people out of extreme poverty with dignified jobs.

Greetings from Bangalore! As one of Upaya’s Senior Associates, I have a number of different responsibilities at the organization. My time is split between donor relations and impact management, and the latter – which includes survey design, collecting data from jobholders, and analysis – has easily become one of my favorite aspects of my role at Upaya. I get to work closely with the data we receive from our partner companies (our investees) which has given me insights into the significant impact we’re having.

Based in our Seattle office, my role at Upaya is generally behind the scenes. But this week, I’m thrilled to be in the field with our team in India kicking off Upaya’s 2019 data collection! We’re halfway through conducting surveys with the employees of Saahas Waste Management, one of Upaya’s partner companies, and I’ve already learned so much about our data collection process.

The Importance of Data Collection

The previous post in the Upaya Fundamentals blog series discussed the elements of a job that we consider important for lifting families out of extreme poverty: a job must be predictable, dignified, and provide income sufficient to move a jobholder out of poverty. We are set on our mission to create dignified jobs and feel confident in our ability to identify and invest in companies who share that goal. We, however, go beyond simply counting the number of jobs our partners create. We want to hear from the jobholders themselves about how these jobs are impacting their lives.

Assessing the impact of our partner companies – and thus the impact of Upaya – is a core component of the work we do. We collect data through multiple different channels. We ask for quarterly performance data from our partner companies and conduct in-person surveys with employees at those companies, which helps us gather both quantitative and qualitative data.

So why do we care so much about collecting data, evaluating the results and sharing it? Because it’s extremely important to us to ensure that truly dignified jobs are being created and that the families impacted by our work are in fact making their way out of extreme poverty. Remaining transparent in our work and data analysis allows us to continually learn and fine-tune our model.

How We Measure Impact

Are the lives of our partners’ jobholders really improving as a result of their employment? Are they receiving a more stable income than they were prior to employment there? What are they doing with their income? Are they satisfied with their jobs? By conducting jobholder surveys, these are just some of the questions that we’re trying to answer.

More specifically, jobholders are asked about household demographics, education levels of its members, income and savings, household assets and livestock, housing conditions, meal consumption and job satisfaction.

Each question has been thoughtfully selected and serves a purpose within our survey. A subset of the questions align with the Poverty Probability Index (PPI), which is a tool used to measure and validate poverty levels. Other questions are guided by the UNDP Sustainable Development Goals and others are based on Upaya’s own interest and indicators for dignified employment.

The income questions are perhaps the most important and most complicated part of our survey. This section asks jobholders to report their current and past income. It sounds simple, but imagine being asked to report the specifics of your current income, your previous income and your spouse’s income! We work with enumerators who are trained to ask these questions carefully and to help respondents answer to the best of their ability. We’ve learned – through our own work in defining a job and through trying to gain a full understanding of a household’s income – that employment is not black and white. Many jobholders and their households work informal side jobs; they collect pay sporadically, and don’t have clarity on exactly when and how much they get paid. Our survey asks respondents to parse out each separate source of income, so that we can learn as much about a household’s entire income stream as possible.

Our Data Collection Process

Upaya’s goal is to conduct three surveys with all of our investment partners – a baseline, midline, and endline survey. We recognize that it can take time for a job to become stable and to receive monetary raises and promotions. Conducting surveys at different points in time helps us capture the results of longer-term employment.

With the help of an outside data collection company, each partner survey exercise takes 2-6 days to conduct, and often requires traveling to partner sites in remote locations. We ask to survey a sample of employees who hold different positions at their companies, but otherwise selection is random. Depending on the size of the company, we will conduct between 50-250 jobholder surveys with each company. Each survey takes 10-12 minutes to administer.

Historically, we’ve worked with paper surveys, which were filled out by hand and manually entered into a spreadsheet. Through trial and error, we’ve learned a lot about surveying jobholders effectively. One takeaway was that upgrading to a mobile survey platform might improve efficiency and accuracy of data collection – we are excited to be rolling out our mobile format this week! Not only is it making data collection much easier, but we’ll be able to view the complete dataset almost immediately after surveys are conducted. Another takeaway was to keep surveys short to maintain respondents’ attention and respect their time. Oftentimes, jobholders must step away from their work to be surveyed, so keeping surveys straightforward and easy to implement is a must.

Putting Our Findings to Work

Once we receive the full dataset from each company, we go through the process of cleaning the data and getting it ready for analysis. The final product then gets published to the Impact Dashboard on our website. It also serves as valuable insight into the effectiveness of our work.

Last year, when I began working on impact management I asked, “what if we receive data that indicates we are not in fact having the impact we want?” This is an important question to raise, because by collecting impact data with the intention of confirming our impact, we must also be prepared to address instances where our mission isn’t being met. For example, we’ve had a few cases where we found that households were already earning incomes well above the extreme poverty line when they started jobs with our partner companies. They are still poor, and we realize it is unrealistic for 100% of the jobs our partner companies create to be reserved for the extreme poor. But if we observe that more and more better-off households are being recruited for jobs, we have conversations with our entrepreneurs about the reasons for this, and how we may be able to steer focus back to hiring the most marginalized.

Upaya is committed to our mission and are quick to raise concerns with our partners. We happily work with them to improve their impact and wouldn’t hesitate to reconsider a partnership if we saw their impact going in the opposite direction. Over the years, we’ve learned the importance of working with entrepreneurs who value impact from the get-go. When we share this mission, we can work together to ensure that our combined efforts have the impact we both strive for.

The Big Picture

Upaya hopes that others in the field can learn from our efforts in impact assessment. It can be daunting to think about creating and conducting surveys, collecting data, analyzing data, and potentially adapting programs based on results. Sounds like a full-time job, right? But it doesn’t have to be. Impact assessments can range drastically in scale, from full blown randomized control trials to simply asking for candid feedback from program participants or beneficiaries. For any organization, we would argue that holding yourselves accountable for the work you’re doing is equally important as the work itself.

Upaya has made impact management a priority, and we look forward to continuing to develop and strengthen our own work in this space. This week, we’re in the midst of our first round of surveys for the year with Saahas. Stay tuned for an updated impact dashboard later this summer, as we plan to conduct three more surveys in the coming months!