By Giovanna Montagner, Gabriela Pérez Yarahuán, Erick Herrera Galván, Karla Pinel Valerio and Oliver Peña Habib

We would like to contribute to the experience exchange taking place in this community of practice by sharing with you the evaluation process that the Center for Learning on Evaluation and Results Latin America and the Caribbean (CLEAR LAC)  has carried out in collaboration with the Walmart Mexico Foundation and its partner organisations for the Smallholder Market Access Program. We will also share our thoughts on using qualitative methods to capture the perspectives of program participants on the change processes in which they participate.

The context

The agricultural market in Mexico is characterised by its structural exclusion of smallholder farmers, contributing to persistent high levels of rural poverty and an uncompetitive market with major inequalities between stakeholders in the value chains and a dependency on intermediaries who take a significant proportion of the final product price.

The intervention

The program works to facilitate the access of small fruit and vegetable farmers to these value chains by offering services that integrate training and development of market linkages for farmers organizations in different states of Mexico. These services aim to increase farmers’ income, facilitating improvements in both the quality and quantity of produce. They also help them formalise farmers organisations and link them to the supply chains of large-scale purchasers without relying on intermediaries.

The programme, run by TechnoServe since 2011, works with three types of partners, who all contribute to meeting these aims:

• • Agri food businesses and large scale retailers who partake in the program by buying directly from the small holder farmers groups;
  • “Social-purpose” intermediaries who help arrange sales at fair prices for farmers lacking the conditions or the interest to organise themselves;
  • NGOs and businesses who receive advice and support from the program to replicate or adapt its model.

The evaluation

Between 2018 and 2020, CLEAR LAC conducted a phased evaluation process to support the scaling-up of the intervention model with a focus on market transformations.

We realised very quickly that traditional evaluation approaches would be limited in how successful they could be in fulfilling this aim and that it would be necessary to work in a participatory manner with program managers and staff. We therefore held a workshop to draw up the evaluation agenda and questions in a collaborative manner, maintaining constant dialogue throughout the evaluation process.

One of the stages of this process was a qualitative outcome evaluation, aimed at analysing the program’s contribution to transforming the Mexican agricultural market so that it could become more inclusive, resilient and sustainable. In order to do this, we turned to program participants and gathered their perspectives on the changes that the programme had helped bring about, whether positive or negative, intentional or not, direct or indirect.

Using an adaptation of Outcome Harvesting, we conducted telephone interviews with the leaders of participant farmer group about the most significant changes that they had experienced thanks to the program, how their perception of themselves had changed both as individuals and as a group , what their five-year vision was, what challenges they still faced and what additional support they needed. Following this, we did a field visit to a sample of five farmers groups to reconstruct their change pathways through a dynamic timeline exercises with the groups’ leadership teams, individual interviews with farmers and non-structured observation. We also held semi-structured interviews with representatives from implementing organisations, “social-purpose” intermediaries and purchasing departments from participant companies. In these interviews, we asked them what they are doing differently and about their thoughts on the program’s intervention model.

By doing this, we were able to identify the progress and areas of opportunity of the program from the perspectives of the farmers and other stakeholders, as well as to facilitate their use as an input for decision-making regarding the program’s future. This process led to a few surprises.

For example, one of the changes most frequently reported by the farmers was the way their organisations were strengthened with the program’s support. This was despite the fact that this was not one of the formal components of the initiative but rather informal advice provided by field staff. Based on this finding, we discussed with the program staff how to systematise the experience of the field staff supporting farmers organizations and create a component especially dedicated to this, which would benefit the farmers that will join the programme in the future.

The evaluation process ended with a series of working groups that had the aim of strengthening the intervention model and planning the next stage of the program based on the evaluation findings and recommendations. The participants included the program’s staff as well as representatives from partner organisations.

This experience taught us the following lessons:

1. Opening up spaces for dialogue and listening can be an entry point for participatory evaluation. Full participation in all evaluation stages was not possible as the program does not operate with a participatory approach. Furthermore, there were logistical challenges due to the geographical distribution of farmers groups and also security issues. However, it was the first time that the voices of the smallholder farmers were heard systematically and factored into decision-making processes. This was not only valuable for the program, but also for the farmers themselves, some of whom shared that they had really appreciated the fact that their opinion had been taken into consideration. The same thing happened as we interviewed implementing organisations, purchasing companies and “social-purpose” intermediaries.

2. The different perspectives of stakeholders are key to evaluating complex initiatives with transformational aims, as they shed light on factors that facilitate or hinder change processes in different contexts. By doing so, it is possible to identify opportunities to build alliances and diversified intervention models.

3. Qualitative methodologies are essential to the evaluator’s toolkit but often their use is limited to extracting information in the data collection phase. To avoid this, evaluation designs need to include “process uses” and establish how the evaluation findings will be fed back to the participants.

4. The qualitative evidence resulting from participatory processes can be helpful in explaining the findings from quantitative evaluations.This study used data and findings from a quantitative evaluation that the programme had already carried out internally. We used this to analyse the changes that the farmers groups had experienced with a rigorous analytical sampling, which allowed us to understand why some groups had achieved greater results in improving their income, productivity and adopting good practice. The findings derived from dialogue with the farmers groups allowed us to understand the interaction of various factors in their change pathways, which explained the diversified results that were observed in the quantitative evaluation.

5. Finally, this experience changed our perspective regarding how to evaluate. Our previous experience focused on the external evaluations of governmental programs, which in Mexico use standardised terms of reference, often with limited possibilities for participatory processes. In this evaluation, instead, CLEAR took on the role of “learning partner”, supporting the programme all the way from the definition of the evaluation objectives and questions to the use of findings and recommendations. This also meant that the evaluation was more useful as the process was adapted along the way according to the changing needs of the program.

We hope you find these lessons useful and we invite you to share your comments and thoughts in the Virtual Forum or in the comments section below. We are especially interested in hearing your thoughts on these questions:

What opportunities are there for linking participatory approaches to other methods and approaches, for example, complex system perspectives and quantitative evaluations?

In your experience, what potential does participatory evaluation have to facilitate transformative initiatives?