This week we feature another guest blogger... Step forward Kieron Kirkland from Nominet Trust in the UK. Below are his latest thoughts on some of the problems that come with shared measurement practice. Here at the Global Value Exchange, we believe his arguments provide further fuel to our aims of empowering stakeholders and promoting the outcomes and indicators from the 'bottom up'.
By Kieron Kirkland
There's been a lot of focus on shared measurement of impact recently. In some ways shared measurement approaches can be really useful. They can help organisations identify robust indicators for success and easily communicate and integrate their impact measurement with wider initiatives. However, as with underwear, I feel a ‘one size fits all approach’ can be very problematic for a number of reasons, particular for early stage and developing interventions.
One challenge of standardising measurement is aptly described by ‘‘Goodhart’s Law’ - roughly that when a measure becomes a target, it ceases to be a good measure. For one thing, people may try to game it. But more concerning is that standardised measurement can skew organisations' missions and end up making them perform to external measures rather than using measures which represent the actual social value they are trying to generate. This is problematic. New approaches won't neccessarily align with these shared measures,especially in the early stage. This can result in new models being deemed a failure before they have a chance to develop because they don't match up. Worse still they can change their approach and suffer mission drift by focussing on external targets.
Linked to this, standardised measures risk putting the focus of measurement on external accountability and not on internal learning and development of practice. If we just focus on external accountability for measurement we are unable to create more nuanced metrics to ensure the continual development of alternative approaches. As mentioned above this is particularly important when developing new innovations, and especially if we're looking at agile and iternative product development.
Thirdly, standardised approaches cannot account for factors we haven’t understood yet. This is especially difficult when scaling or growing as we encounter novel situations in new and dynamically changing environments. Measuring only for what we know limits our ability to adapt or sometimes even see these unaccounted for factors. This can lead poor or inappropriate implementation. I've taked more about this here.
Lastly, we have to remember that we all see change differently. One organisation's theory of change may be different to another. Metrics represent and reinforce specific types of social value and theories of change. Neccessarily that means that standardising measurement means standarising theories of change. That’s fine if we all agree on one theory of change, but in many cases we don’t. And for good reason, the world is very big and very complex.
Standardised measures have a useful role to play in communicating the social value of our work to others and enabling us to integrate our work with other intiatives. However, equating evaluation and measurement with standardised measures is simply wrong. Doing so risks eradicating the value that can come from it in the first place.
- See more at: http://www.nominettrust.org.uk/knowledge-centre/blogs/some-problems-shared-measurement#sthash.dHjOpyjo.dpuf
Nominet Trust is the Uk's leading social tech funder. We bring together, invest in and support people making inaginative use of technology to address complex social challenges.