Lessons from using a Theory of Change in a Participatory Evaluation

by Viola Cassetti and Joan J. Paredes-Carbonell

Our professional paths crossed in Valencia in 2016, when Viola was finishing her European Master in Public Health (EuroPubHealth) and about to start her PhD at the University of Sheffield (UK) and Joan was working as Deputy Director General of Health Promotion at the Valencia regional health authority.

Our first project was to adapt the NICE guidelines on community involvement to the Spanish context using a collaborative approach. We spent two years co-coordinating a group of more than 80 professionals who actively participated in the project. You can acess the guide (only in Spanish) by cliking on it (Cassetti et al., 2018).

Since then, we have continued to work towards the same goal -both together and on parallel projects- of encouraging community participation at every stage of health programmes. Our shared knowledge and experience led us to embark on our first participatory evaluation of a health promotion initiative, the ‘La Ribera Camina’ programme, which promotes physical activity through group walks with the accompaniment of health and physical activity professionals.

La Ribera Camina was piloted in 2019 in two municipalities in the province of Valencia (Spain). Today, the programme runs in 11 health districts and 23 municipalities of La Ribera administrative division, south of the city of Valencia (Egea Ronda et al., 2022). Initially, the programme’s structure was largely defined by officials in a top-down decision-making process, but Joan and other colleagues suggested a redesign of the programme to incorporate a bottom-up approach informed by perspectives that emerge from community participation.

As we discussed in our recently published chapter in the Global Handbook of Health Promotion, Vol. 1: Mapping Health Promotion Research (Cassetti and Paredes-Carbonell, 2022), the evaluation of intersectoral actions and community health programmes is particularly challenging due to the complexity involved in both. There are multiple stakeholders who interact with one another in different ways. These interactions generate various dynamics and outcomes that are difficult to anticipate or predict. This is why we comment on the importance of adopting a systems thinking approach. Systems thinking helps us understand initiatives as embedded in their contexts, rather than as a set of isolated activities. In other words, systems thinking helps us analyse the complexity involved in initiatives and how they interact with the contexts in which they are implemented, which are -in their own right- complex. Hence the importance of the participatory approach, because when we involve the beneficiaries in the evaluation of a programme, it allows us to capture the different dynamics the programme is generating in each context, and to see how the programme interrelates with these contexts.

Thus, we employed both a participatory and systems thinking approach during the process of designing the theory of change. In the theory of change method, we create a highly visual diagram for participatory planning and evaluation. The diagram is the output resulting from dialogue between the different stakeholders and it should explain why and how they believe the intervention can be successful (Cassetti y Paredes-Carbonell, 2020).

A theory of change visually represents how a programme works. It identifies the different components and how they relate to one another, the aims and desired outcomes of each component and the actions with which they hope to achieve them. It is similar to -but goes beyond- the logical framework, which also invites participants to reflect on why they expect to achieve these changes and the factors that can help or hinder these processes.

Four theory of change sessions were organised with the aim of outlining the theory implicit in the La Ribera Camina programme and identifying how it was performing, areas for improvement and the results it was intended to obtain, so that decisions could then be made on which results should be measured and how. The sessions were directed at different focus groups: health professionals, local government representatives and two sessions for people in the community who take part in the walks. They were recorded with the participants’ consent and analysed using a qualitative method of thematic analysis. A follow-up event was organised to share the results and participants engaged in workshops to prioritise improvement measures for the programme, its evaluation, and the participation process.

This methodological approach -a combination of participatory and qualitative methods and systems thinking- has yielded many unanticipated results. While, as expected, the programme itself was evaluated as the participants commented on what was going well and what needed to be improved (Egea-Ronda et al., 2022), it also went beyond that to identify all the expected change, not only in terms of health but also in terms of organisational and institutional relationships, changes in the programme itself and in the physical environment. For example, on the subject of organisational changes, the three focus groups all agreed the programme had generated an intersectoral dynamic, interinstitutional collaboration and citizen participation.

Moreover, the evaluation process itself became a source of knowledge production. Each participant offered their vision of the programme during the process and all these perspectives were gathered in the final version which set out a new vision for La Ribera Camina, what it involves and how it impacts both the beneficiaries and the individuals working in the programme (Cassetti y Paredes-Carbonell, 2022). For their part, the individuals who participated in the workshops learned a great deal, acquiring a more complete understanding of what the programme involves, helping to produce the theories of change and participating in the final sessions. Each person offered their perspective on the programme and -through dialogue with the other participants- expanded their understanding of La Ribera Camina and its potential impact in the community.

Furthermore, agreements were reached between all the parties involved on the programme’s areas for improvement. These include enhancing and extending the current routes, incorporating complementary activities to the walks such as healthy nutrition workshops, yoga/relaxation techniques, raising the profile of the programme and investing in publicity.

As part of the evaluation, the participants also shared their experiences of how local dynamics inherent to each context impact the implementation of the programme. This generated ‘data’ that fostered a systems thinking perspective when analysing the programme and the areas where it is implemented.

The process required back-and-forth communications between the ‘local’ component (the programme in each municipal area) and the ‘global’ element (the overall health programme). It involved exchanges between professionals and participants, and the results provided evidence that was then used to redesign an improved initiative, which was co-created both in its programme planning and implementation (Cassetti y Paredes-Carbonell, 2022).

To conclude, while the process was lengthy, and had to be interrupted several times because of the COVID-19 pandemic, it was an enriching experience for the researchers and for the programme professionals and participants. It signalled a new way of embarking on an evaluation and we hope it can be replicated in other contexts and scenarios.

References

Cassetti, Viola, Victoria López-Ruiz, Joan J Paredes-Carbonell, and Grupo de trabajo AdaptA GPS. 2018. PARTICIPACIÓN COMUNITARIA: Mejorando La Salud y El Bienestar y Reduciendo Desigualdades En Salud Guía Adaptada de La Guía NICE NG44: «Community Engagement: Improving Health and Wellbeing and Reducing Health Inequalities». Guiasalus.es.

Cassetti, Viola, and Joan J. Paredes-Carbonell. 2020. “La Teoría Del Cambio: Una Herramienta Para La Planificación y La Evaluación Participativa En Salud Comunitaria.” Gaceta Sanitaria 34 (3): 305–7. https://doi.org/doi.org/10.1016/j.gaceta.2019.06.002.

Cassetti, Viola, and Joan Paredes-Carbonell. 2022. “Participatory Approaches to Research Intersectoral Actions in Local Communities. Using Theory of Change, Systems Thinking and Qualitative Research to Engage Different Stakeholders and Foster Transformative Research Processes.” In Global Handbook of Health Promotion, Vol. 1: Mapping Health Promotion Research, edited by Didier Jourdan and Louise Potvin, 1st ed. Springer, Cham. https://doi.org/10.1007/978-3-030-97212-7_25.

Dirección General de Salud Pública, Joan Paredes, and Viola Cassetti. 2019. “Participació Comunitària En Salut.” Viure En Salut. Vol. 115. Valencia. http://www.sp.san.gva.es/biblioteca/publicacion_dgsp.jsp?cod_pub_ran=948023424&tacc=17.

Egea-Ronda, Ana, Montserrat Niclos Esteve, Amparo Ródenas, Mariví Verdeguer, Viola Cassetti, Carlos Herrero, José M. Soler, and Joan J. Paredes-Carbonell. 2022. “Teoría Del Cambio Aplicada Al Programa de Promoción de La Actividad Física ‘La Ribera Camina’. [Theory of Change Implemented in the Program to Promote Physical Activity ‘La Ribera Camina’].” Gaceta Sanitaria xxx (x): 7. https://doi.org/10.1016/j.gaceta.2022.02.012.