by Sara Vaca
After studying business and spending my early career working in the private sector, discovering what “participation” meant, albeit theoretically, was key. I understood that “participation” meant that several stakeholders had a voice (and vote) in decision making during the four phases of the evaluation: the design, data collection, analysis, and interpretation… seemed simple enough!
From that moment (back in 2012), I began to notice that most evaluation reports included in their summary or methodology something along the lines of: “this is a (highly) participatory evaluation”, and some even mentioned it in the title itself (“participatory evaluation of…”), when what they really meant was that they had consulted many people or groups, but only as informants…! I wondered why they called it participation when what they actually meant was that they had consulted a wide sample.
As is common in many other issues, the problem was a lack of clarity and transparency, largely because it cannot be “seen”… And that is what I love about Data Visualisation (and information) which, despite its shortcomings, leaves less room for confusion.
This led me to think that we could reveal the level of participation by different groups in different phases in the form of a matrix. Based on the idea of “heat maps”, where the different colours in the box indicate higher or lower frequency or intensity, I thought I could represent participation in the evaluation in the form of one of these maps to show how involved stakeholders were in the different phases. And so, the “Participation Scanner” was created.
This image is a typical representation of the vast majority of evaluations I encounter: the donor or organisation designs the basic lines of the evaluation in the Terms of Reference, which are then discussed and slightly modified by the evaluation team during the initial phase. The evaluation team then collects the data and analyses it, and sometimes the findings are validated in a wider workshop to aid interpretation.
However, a truly participatory evaluation should present green cells (representing a significant influence in decision making) for stakeholders such as the communities, local authorities or other groups equally interested in the intervention. Generally speaking, these groups have been underrepresented in evaluations.
I have gone on to refine and rework the Evaluation Scanner, incorporating the dimension of use of evaluation results into the phases of the evaluation process.
From my own experience, in the contexts I have worked in as an evaluator, it is difficult to move “power” towards the columns to the right (local authoroties and communities). I have often tried to suggest doing this but have rarely achieved it due to the extra time, resources and investment required for capacity building. However, in my evaluation reports I always include the scanner in the methodology section to show the levels of participation for each group in each phase and I explain these factors in the limitations… and I never say that the evaluation was “highly participatory.”
