How Do We Know We’re Making a Difference? The Challenges of Measuring Impact

Here at ASC, we collectively work on transforming fish farming towards environmental sustainability and social responsibility. As an organization with relationships across the industry, we can’t just try hard and hope for the best. We want to know whether our work is making a difference, and how we can learn and adapt where we are not. Our partners, stakeholders, and the communities we engage with across the industry want proof of our – and their – impact. This is why we have established our Monitoring and Evaluation Programme.

By Jill Swasey, Head of Monitoring and Evaluation

When we think about ‘impacts’, we want to identify the effect of an action on something. While we want to have a positive impact, it can go both ways. Clear, measureable indicators are developed and tracked to evaluate whether a desired impact is achieved, or not.

Through evaluation, we monitor both quantitative and qualitative information. Quantitative data are the numerical datasets, such as those that we receive through audit reports of certified farms, logo use and certified products across markets. These allow us to ask questions such as “what” and “how many” across countries, farm types, species, etc. These reliable data demonstrate both the growth and reach of our work and communicate the uptake of certification across markets. Qualitative data is descriptive by nature, and generally collected through observation and interviews. While it can be more difficult to analyse, qualitative data can provide a grounding to trends observed, speak to the impacts experienced by actors engaged in the industry, and help us spot things that don’t show up in the numbers. Both forms of information are valuable and using them together helps us complete the picture.

At ASC, we face certain challenges. We can measure how certified farms are performing, but of course what we can’t know is what would have happened without ASC certification. The ASC’s commitment to data and transparency means there are reams of information about certified farms, but the body of information around uncertified farms is far more limited and inconsistent, making comparison difficult. We utilize data on the environmental and social performance of farms against the indicators in our standards, with interviews, and increases in certifications and approved products to communicate how more farms are operating responsibly and farmers and consumers are realizing those benefits. However, we cannot yet compare these gains globally to farms not in our programme; though in some cases, these comparisons can be made for certain production systems and regions.

Another challenge is recognizing impacts at scale. While we want to know our collective impact, there is also tremendous value in reporting impacts across geographies, markets, species, and so on. In aquaculture, we see considerable variability across production systems, regions and climates, and recognize that context matters. This can complicate fine scale evaluation, but these differences can also highlight some more pronounced changes that might otherwise be masked. One of the ways we approach this is by digging into certification reports to understand where non-conformances are raised across social and environmental impact areas. From this information we get a look into which requirements may be more difficult to meet by certain production systems and in certain regions, and therefore require improvements. For example, we can explore which shrimp producing countries required more improvements to meet the standard, and what types of improvements those were.

As the ASC programme grows, we evolve our thinking and attention to what changes we can expect and in what timeframe. The first ASC farms were certified in 2012, and we have now achieved nearly 1,300 certified farms. Realizing long-term outcomes takes time, but we have set the foundation for consistent monitoring to track our progress towards achieving our mission.