The Learning Portal for Design, Monitoring and Evaluation for Peacebuilding

You are here

Discuss

Complexity and Intangibility: No Excuse for Bad Measurement Practices

We in peacebuilding like to think of our profession as uniquely distinct from international development, humanitarianism, and foreign policy and international assistance broadly.  This mentality has rippled down into the evaluation of peacebuilding, where the measurement of results is described as “difficult” (to put it mildly) because of the complex environments and themes that we work in and on, the intangible and oft long-term nature of results, and the non-linearity of change processes. 

While these challenges are very real, the perception of them being unique to peacebuilding is not as correct as we like to believe.  It follows, then, that nor is the challenge of measurement unique to peacebuilding, and furthermore, is not as intractable as is perceived by some.  Could it be that our own perceptions of unique challenges are actually undermining the quest to overcome said challenges?

The theme of this year’s annual conference for the American Evaluation Association was “Evaluation in Complex Ecologies: Relationships, Responsibilities, Relevance”.  Over the course of three days, hundreds of presentations were given by evaluators, consultants and program managers on how complexity affects their work, and how evaluation designs can accommodate for this.  I learned many things—including the use of photography and collaborative art in the evaluation process (look for a blog on this soon!).  But what hit me the most was the realization that complexity, intangibility and non-linearity are very frequently fundamental traits and challenges in the evaluation of other professions and fields. 

Other fields and professions are actively struggling with and overcoming many of these challenges: social psychology, education, policy and advocacy, even international development and many other professions!  We, as a cross-cutting and interdisciplinary field, should at minimum take note of these efforts.  Even better would be to actively track and participate in such efforts, and export the exportable to our own work!

Oh, the things peacebuilders could learn if we only broke out of the professional bubble we have created around ourselves!

Jonathan White manages the Learning Portal for DM&E for Peacebuilding at Search for Common Ground. Views expressed herein do not represent SFCG, the Learning Portal or its partners or affiliates

This phenomenon of creating International Affairs bubbles around certain fields is quite frustrating! Not only does the exclusive grouping hinder information sharing- it creates mindsets of hierarchy. By saying peacebuilding is so different that is deserves it's own set of tools, methodologies and paradigms, it inherently alienates itself from other fields, say Global Health, Infrastructure development or ICT. When these various groups all must vie for funding, international or domestic attention, they find they must prioritize themselves over others, as THE BEST route to furthering global wellbeing. This competition in turn limits overall efficiency that COULD HAVE been gained if everyone could collaborate and share methods, terminologies, and practices.

Oh in a perfect world Hannah!  Well said.  It seems that his conundrum of complexity and intangibility creates a great opportunity for M&E.  Precisely BECAUSE there is complexity it becomes all the more important to have a well designed M&E process with clear, tangible outputs.  The intangibles of peace building sometimes need to be sold and finding creative mechanisms to "sell" this "product" will come DIRECLTY from M&E.  I realize I am making a case for turning intangible into the tangible... but that's the rub isn't it?

But I am preaching to the choir.  I am very interested in the upcoming blog on photography and collaborative art as evaluation tools!  Could you give us a sneak peak?   

Hannah, one of the more interesting developments in attempting to burst this bubble, I think, is the concept of human security. Mary Kaldor and Shannon Beebee recently wrote an excellent book on the subject. Mercy Corps is also perhaps at the forefront of incorporating peacebuilding approaches, values and principles into traditional development activities. Some of their case studies and evaluations are actually available on the Portal here! On the flip side, however, is SFCG, where we frequently integrate development programming towards achieving our peacebuilding goals and objectives. 

You raise an interesting point, Kevin: that good design planning (SMART objectives and indicators, solid M&E plans and appropriate budgeting for the context and needs) is a way of dealing with that intangibility and complexity from the get-go. One of the problems of the way M&E is currently practiced is that there are 'M&E people' and 'program people'. While specialists are certainly needed, what is also in short supply are program staff with a firm grasp and appreciation for M&E. I'd like to give you a sneak peek at the blog, Kevin, but I still have to write it haha. But it will include consistently taking photos of a particular location over time to track changes and analyse the photographs for evidence of change. 

Cheers,

Jonathan

I am often inclined to chuckle when I hear the ambiguous issue of complexity and intangibles brought up in a field of work.  I often find, especially at conferences, that these terms tend to create cover for a host of factors that practitioners and academics cannot seem to articulate well.  So they just lump them all together and say it’s complicated or there are unknown factors involved.  Of course situations are not simple and those in the field can never see the whole picture, but ultimately I’ve come to conclude that many, in whatever area they make their bread, are basing these “difficulties” on their own constructed theories and perceptions. 

And it is those socially and individually inflicted wounds that constrain individuals and organizations to solve great problems. Each of the numerous fields, be it peacebuilding, development, health, education, law, or ‘fill in the blank’, are all placed in a bubble by these perceptions, many of which are insulated and perpetuated within their own institutions.

Monitoring and evaluation, while also in danger of their own constructed viewpoints, can be the key to unlock those mental barriers and the bridge between any number of professional fields. This can be so, because M&E, as I’ve come to see it, is a strongly technical discipline that can maneuver through the mess of complex and unforeseen factors to find the causes for these difficulties and even provide helpful lessons learned.  It would be easier, of course, if all those involved, M&E specialist, program staff, field staff, managers, and executives, would commit to a strong and SMART M&E practice.

Hear hear, Kyle! Indeed, as another colleauge pointed out in another discussion (http://www.dmeforpeace.omnidev3.com/discuss/monitoring-practice-point), the entire process of DM&E is meant to help manage the complexity and account for, to some extent, the unknowns. And of course, in such complexity and unknowns, any plan must be inherently flexible and 'quick on its heels' if you will. 

As for the capacity of staff, but particularly programme staff, to conduct high quality M&E, there are, thankfully, many efforts underway. Perhaps one of the larger and more concerted efforts is by UNICEF and their push for 'equity-focused evaluation'. They're currently conducting a whole series of webinars and self-guided courses to build capacity in this area. 

Complexity is a theme that's coming up in nearly every profession in the world today. Or at least those that have their eyes open to what is happening.  Part of the complexity is the scale of problems that we're facing, part of it is the amount of available information and data so that problems seem bigger. And a large part of it is how very few problems truly stand in isolation - the degree of interconnectivity continues to increase again both in reality (more connections through digital media) and also in our awareness of connections that have always been there, but largely ignored. 

How professions are responding to this challenge varies, but there are underlying prinicples that are common. Complexity is something that can be studies, and as another comment points out, it does have something to do with things at the edge of our current knowledge. What's complex today, becomes simple in a few years. Or at least that's what we can hope for in the progression of knowledge.

One organization that studies and writes about complexity and has influeced many in the Software Development field (which I'm also part of) is David Snowden. You can learn more about him and the Cynefin framework here: http://cognitive-edge.com

Best,

Evan