By Juan Carlos Sanz and Esteban Tapella
The order of events in participatory evaluation does not stray far from that used in conventional evaluation. And yet, there are some clearly differentiating elements and steps that we need to be aware of if we want more actors to participate.
The planning phase is particularly relevant. At this stage, the evaluation team identifies which actors are going to participate in the process; determines the topics and questions which the evaluation intends to answer; selects where and how the necessary information will be gathered; creates spaces for reviewing and analysing the data; establishes the best formats for communicating the evaluation results; and chooses the most appropriate mechanisms to ensure that these results are put to use. The work plan also specifies the timeframe for the evaluation and estimates what resources will be necessary for its implementation.
Within the planning phase, the formulation of the questions is perhaps the most important activity. The entire evaluation process depends on these questions, especially the search for the right information needed to answer them, and thereby generate useful recommendations that can lead to improvements to the intervention.
EVALUATION QUESTIONS
Evaluation questions are formulated as a way of guiding the evaluation process and responding to the requests for information that have been prioritised for the programme or project.
It is important to be aware that good evaluation questions: (1) correspond to a need for information or the identification of a solution; (2) refer to issues that can indeed be addressed by the evaluation and do not correspond, for example, to the follow-up system of a programme or an audit.
Participatory approaches specifically seek to make the most of the knowledge of local teams. It is therefore essential to adapt the instruments so they can be used by non-professionals who may be unfamiliar with the subject. When participants understand the logic of the evaluation better and are more familiar with the tools, the positive effects are twofold: the evaluations will be of a higher quality and the evaluation team will garner more knowledge.
A common problem in evaluation is that the questions are too broad and can therefore be interpreted (and answered) in several different ways. The risk is that the answers generated do not correspond to the type of evidence sought when the question was formulated. One way of avoiding this problem and identifying more concise topics and questions is to hold a session where the entire team can participate and explore everyone’s ideas in depth. This led us to create the game, ‘What Shall We Evaluate?’
A Game to Encourage Thinking
Games designed to encourage thinking are intended to transcend mere entertainment. In other words, the game is designed to comprehensively address a complex topic, such as the formulation of sensitive and deep questions that will guide the evaluation process in a participatory, non-hierarchical and non-imposing manner. The game includes a board (to be printed in A3 format, both sides) and an instruction manual. You will also need a dice and some blank cards to write your questions on.
Objective
The objective of the game is for the participants to decide how to formulate the evaluation questions together.
The game can be made more exciting with these two challenges:
-
-
- Personal challenge: each player has to place all their cards on the board.
- Collective challenge: by the end of the game (when all players have placed all their cards on the board), each of the boxes numbered one to six have at least two cards assigned to them.
-
Instructions
– Place the board face down on the table without reading the information and then give each player three green cards and two pink cards.
– Each player has ten minutes to think of three questions that they would like the evaluation to answer on the intervention. They should write one question on each of their green cards and place them face down so that the other players cannot see them. The two pink cards are saved as wildcards.
– Now, turn over the board and read out the information from each of the boxes, ensuring that everyone fully understands the meaning.
– The first participant rolls the dice to select one of the six boxes on the board (the number on the dice matches the number on the board). If the participant thinks that one of the questions on their green cards corresponds to the text in the box, they should read it out to the group.
– This will then lead to a group discussion in which one of three decisions is made:
- The question DOES correspond to the box:
In this case, the participant can place the card next to the corresponding box on the board and the dice is passed to the person on their right.
- The question DOES NOTcorrespond to the selected box, but it DOES correspond to one of the other five boxes:
In this case, the participant should save their card for a future round.
Before passing the dice, the participant can choose between two options: (1) use one of their wildcards to formulate a new question that does correspond to the selected box (and check it is accepted by the group); (2) not use a wildcard and ‘open up the turn’ to the rest of the participants so one of them can put down one of their green cards (as long as the group agrees that it corresponds to the box that was selected previously by the dice).
The dice is passed to the player on their right.
- The question DOES NOT correspond to ANY of the boxes on the board.
In this case, the participant will place their card in the box marked X and pass the dice to their right.
– At the end of the game, remember to keep the cards in the piles they were assigned to.
In the following video (with English subtitles), Juan Sanz talks about the origins and process of the game and explains different ways it can be used. He also puts forward some recommendations for how to adapt it to different groups and contexts.
