by Ana Manzano
Over the last decade, there has been a growing interest in involving citizens in the evaluation of policies, programmes and social interventions. In my article ‘User and Stakeholder Involvement in Realist Evaluation’, published by the Laboratory for Interdisciplinary Evaluation of Public Policies (LIEPP, Sciences Po, France), I explored the theory, methodology and practice behind the participation of users and stakeholders in evaluation. I specifically focused on the realist evaluation approach and compared it with other classic evaluation methodologies such as collaborative, participatory and empowerment evaluation.
The article begins by exploring some terms typically used in participatory approaches and seeks to enhance conceptual clarity by highlighting similarities and differences. It then explores participatory approaches used in realist evaluations through a literature review of the participatory methodological strategies most commonly employed within this evaluation framework. I identify three principal models of engagement: vague, targeted and integrated before comparing the realist evaluation approach with collaborative, participatory and empowerment evaluation. I note that in most realist evaluations the evaluator is in charge and determines the study design and data collection methods, leading to a power imbalance. The article concludes that the principal purpose of realist evaluations (even those at the more participatory end of the spectrum) is to establish how the programme works in different contexts. In other words, the evaluation objective focuses on assessing programme efficiency rather than empowering the stakeholders involved in the evaluation.
Realist evaluation
Popularised in the late 20th century, mainly by Pawson and Tilley (1997) in Europe and Henry et al. (1998) in the United States, realist evaluation is a theory-driven approach (Chen, 2012) that focuses on understanding causality in complex policies, programmes and social interventions. It involves investigating how results and mechanisms interact in different contexts. Realist evaluations do not respond to the typical evaluation question: ‘Does this programme work?’.
Rather, they ask: ‘How does this programme work, for whom, under what circumstances and why?’. Thus, the context of the programme is put at the centre of the evaluation and the key programme stakeholders are the principal actors who can help untangle the complexity of the intervention.
This evaluation approach clearly sets out the ‘programme theories’ (assumptions and expectations, hypothesis of how it will work etc.) enabling a better understanding of the complexities involved in its implementation. Realist evaluation also incorporates social science substantive theory into its analysis in order to enhance the rigour of the evaluation.
However, the incorporation of participatory elements into the design and data collection methods of realist evaluation has also brought about a lack of conceptual clarity, constituting a significant challenge. A clear distinction is not always made between participation as a means of collecting data and participation as a goal in itself to achieve empowerment among the communities involved in the evaluation.
Citizen participation models in realist evaluation
Collaborative, participatory and empowerment evaluation, co-production and action research are examples of evaluation approaches that involve citizen participation. The premise underlying these approaches consists of involving citizens and stakeholders in the programme not only as study subjects but also as active collaborators in the evaluation process. However, citizen participation in evaluation practice has been the subject of much debate. While some see participation as a way of making evaluation more democratic and social programmes more useful and relevant for communities, others question the effectiveness and impartiality of the results of participatory methods.
The article published in SciencesPo LIEPP mentioned above included a scoping review of participatory methodological strategies used in realist evaluations, revealing three principal models of engagement: vague, targeted and integrated. Each of these models represents different degrees and manners of involving citizens in the evaluation process. In the vague model, citizen participation is limited and not clearly defined, often leading to only a superficial inclusion of their perspectives. In the targeted model, citizens participate in the evaluation in specific phases but the evaluation design is largely decided by the evaluator as are any decisions regarding data collection methods and the way the research and analysis are carried out. There are fewer studies on the integrated model, in which there is genuine collaboration between evaluators and end-users who define the data collection and interpretation methods together.
Comparison of the approaches: realist evaluation and conventional participatory evaluation methods
By comparing realist evaluation with conventional participatory evaluation methods such as collaborative, participatory and empowerment evaluations, it becomes clear that in most realist evaluations, the evaluator maintains their position of authority, deciding the data collection and analytical methods used.
This contrasts with methodologies that seek to put citizens at the centre of the evaluation process, ensuring not only that the evaluation is effective and relevant, but also that it empowers the participants. My article reveals a tension that is inherent in realist evaluation between the desire to incorporate citizen participation and its practical application, which does not always prioritise empowerment even though it includes citizens in the evaluations.
In conclusion, realist evaluation is a valuable framework for understanding the complexities of social programmes and policies and can incorporate citizens into the phase of evaluation design. However, its potential to promote genuine and empowering citizen participation remains a work in progress. The challenge for realist evaluators lies in how to adopt more inclusive and equitable approaches without compromising the rigour that characterises this evaluation research.
references
Chen, H. 2012. Theory-driven evaluation: Conceptual framework, application and advancement. Evaluation von Programmen und Projekten für eine demokratische Kultur. R. Strobl, Lobermeier, O., Heitmeyer, W. Wiesbaden, Springer VS
Dossou, J.P. and Marchal, B., 2020. Evaluación realista. Evaluación de las Intervenciones Sanitarias en Salud Global.
Fetterman, D., Rodríguez-Campos, L., Wandersman, A., O’Sullivan, R.G. 2014. Collaborative, participatory, and empowerment evaluation: Building a strong conceptual foundation for stakeholder involvement approaches to evaluation (A response to Cousins, Whitmore, and Shulha, 2013). American Journal of Evaluation, 35(1): 144-148
Henry, G. T., Julnes, G., Mark, M.M. 1998. Realist evaluation: an emerging theory in support of practice. New Directions for Evaluation 78: 1-109.
Manzano, A., 2024. User and Stakeholder Involvement in Realist Evaluation. LIEPP SciencePo
Pawson, R., Tilley, N. (1997). Realistic Evaluation. London, Sage.
