Beyond the numbers: overcoming challenges in measuring impact
For many of the nonprofits and NGOs we work with, measuring impact is a critical component (and ultimately proof) of an organisation's success. Accountability and transparency has always been important for funders as they seek evidence of effectiveness and measurable change, but in this era of political uncertainty and global conflict, effectively illustrating impact has become even more crucial.
However, as many of us who work in this sector know all too well, proving effectiveness isn't always straightforward and is often an ongoing challenge.
The challenges
Defining Impact
One of the first obstacles organisations encounter is simply defining what impact means in their context. Is it more easily quantifiable metrics, such as the number of people reached by their programmes or volunteer participation? Does it include policy changes, which may take significant time to fall into place? Or is it something less tangible, like increased community resilience or improved quality of life?
Develop a theory of change. A tried and tested method of supporting the definition of impact is to produce a clear ‘theory of change’. This visual representation of how your activities lead to desired outcomes can give focus and a framework for defining what a successful outcome is for your organisation. For instance, the Health Data Collaborative, a World Health Organisation initiative we have worked with over the years, created a theory of change linking their work to improved health outcomes. This clarity helped them define specific, measurable metrics that they could then reflect out to their stakeholders.
Whether you're a small grassroots organisation or a large international nonprofit, a well-developed theory of change helps test assumptions, identify gaps, and create a shared understanding of how change happens, and how impact can be measured.
Data collection hurdles
Once you've defined your impact, how do you gather the data to prove it? Limited resources, hard-to-reach populations and ensuring data quality, are just a few of the obstacles organisations can face in collecting data.
Collect as many different kinds of data as you can, both qualitative and quantitative. Although this can make it more difficult to have aggregated or comparable data for reporting and visualisation, it does make it possible to wrap your data around real stories and narratives.
An example of this are the multi-year impact reports we created for The Geneva International Centre for Humanitarian Demining (GICHD). The GICHD had used a mix of data collection points, from video testimonials and interviews with stakeholders, through to data around their publications, resources, number of training courses completed and support broken down by country.
Resource constraints
Many of our clients do not always have the reserves (both funding and personnel) to dedicate to data collection or impact measurement. But that doesn't mean it's impossible.
Leverage partnerships. Our work with Leonard Cheshire and Sightsavers in producing a Disability Data Portal included collaborations with other organisations who provided data that we could then use alongside existing datasets. Another of our clients, the United Nations working group The Washington Group on Disability Statistics, were also able to provide context to the data collected under their methodology.
Although data collected by different methods can lead to issues in comparing or amalgamating data, as long as this is understood and data is not represented in a way that can cause bias or mislead, finding partners who can provide supporting data is a low effort method of adding resource to your data collection and impact assessments.
Stakeholder expectations
Different stakeholders often have different ideas about what impact should look like and how quickly it should be achieved. Balancing these expectations while maintaining integrity in your reporting can be a delicate situation.
Use data visualisations. Another piece of work we completed for the World Health Organisation was to support them in the creation of an interactive online data dashboard that allows stakeholders to explore impact data at their own pace. This increased transparency and engagement, while also educating stakeholders about the complexities of the work.
Common pitfalls
Like any complex process, impact measurement has its share of potential missteps. One frequent issue is the over-collection of data - gathering information simply because you can, rather than because it serves a purpose. This not only wastes valuable resources but can obscure meaningful insights beneath unnecessary metrics.
Another challenge is the temptation to focus solely on positive outcomes. While it's natural to want to highlight the success of your organisation, understanding where interventions fall short is equally valuable for learning and improvement.
External factors can also influence your results, yet they can be overlooked in impact assessment. For example, improvements in community health might be partially due to a government initiative rather than your program alone. Being transparent about these external influences strengthens, rather than weakens, your impact reporting.
Be cautious about making comparisons without proper context. While benchmarking against similar programs can be valuable, differences in local conditions, resources, and target populations can make direct comparisons misleading. Instead, focus on demonstrating progress against your own objectives.
In summary
impact measurement will always be a unique challenge as every organisation differs in their metrics and what data is available to them. While starting small with an iterative approach allows you to refine your methods over time, it is also important to remember the human element in impact measurement, and bringing along your beneficiaries or audience is key in this respect.
Using qualitative participatory data collection methods alongside quantitative methods can make sure voices are heard and a human narrative can be wrapped around your impact data. implementing control user groups to better understand the true impact of your interventions also helps to involve your audience in the process.
Perhaps most importantly, impact measurement is an ongoing process rather than a one-time assessment. Build data collection and feedback mechanisms into your programs from the start, rather than treating them as an afterthought. This approach not only makes the process more manageable but also allows you to continuously improve your programs, adapt to changing needs, and strengthen your case for funding.
With over two decades of experience helping organisations transform complex data into compelling evidence of change, we can help you tell your impact story more effectively. Get in touch to discuss how we can help.
-
Owen Priestley
Strategy Director
