Reflections on Artificial Intelligence and the Future of Participatory Evaluation
by Angie Pereira Calvo
A few months ago, I sat for more than two hours with a group of women, listening to their experiences with a social program. There was no recorder running. I held no questionnaire. There was silence, pause, chocolate, and a trust that had taken weeks to build. In that conversation, I learned more about the real determinants of access to health services than from any dataset I had ever processed.
I genuinely wonder what part of that conversation an artificial intelligence (AI) system would have captured. And I also wonder what part of it we would have sacrificed if the methodological emphasis had been placed on the efficiency of data processing rather than on the quality of listening. This, in my view, is the central question the Latin American evaluation community must ask itself right now: before the adoption of AI tools silently redefines what we mean by participation.
Participatory evaluation was not born as a method. It was born as a political stance: born of the recognition that conventional evaluation models reproduced asymmetric power relations. Participatory evaluation promised to invert that logic: it promised that those who live the effects of an intervention also have the right and the capacity to name, interpret, and judge them. It promised that evidence production could itself be an emancipatory process, not merely a technical input for other people’s decision-making.
It is a promise that, honestly, evaluative practice has fulfilled unevenly. Many processes labeled as participatory have been, at best, well-intentioned consultations; at worst, exercises in legitimizing decisions already made. But the promise remains valuable. And it is precisely that promise that is under tension today.
“Participation is not a method that is applied. It is a relationship that is built. And relationships cannot be scaled without being transformed”.
The promise of artificial intelligence applied to evaluation is seductive: and, in part, legitimate. In a context where development cooperation budgets are contracting (OECD, 2025), and where increasingly more evidence is demanded with fewer resources, AI offers something difficult to refuse: greater analytical capacity, faster processing times, and the ability to synthesize volumes of information that are unmanageable manually.
Tools such as large language models can process hundreds of policy documents, transcribe interviews in multiple languages, identify thematic patterns in qualitative data, and generate draft findings matrices. These are genuine capabilities that, well used, can free up evaluators’ time for what matters most: critical analysis, judgment, and dialogue.
The United Nations Evaluation Group (UNEG) acknowledged this reality at its 2025 General Assembly, approving a set of ethical principles for the use of AI in UN system evaluations. It is a meaningful step forward. But principles alone do not resolve the underlying tension.
The problem lies in the type of participation that AI can process. AI systems learn from data. And the data that exists in sufficient volume to train models represents, overwhelmingly, certain forms of knowledge: knowledge that was written, transcribed, digitized, published, indexed. Knowledge that circulated in languages with a massive internet presence. Knowledge produced under the conceptual categories dominant in academia and in the international development system.
The knowledge that emerges from a two-hour conversation with women living in a rural community is not in those databases. Not because it is less valid, but because it was never designed to be processed at scale. Because its power lies precisely in its specificity, its contextuality, its resistance.
When an AI system analyzes qualitative data, it identifies patterns. And patterns are, by definition, what repeats. What is singular: what emerges only once in a specific account of a specific experience: tends to disappear in the analysis. But in evaluation, it is frequently the singular that matters most.
“AI can identify that 73% of interviewees mentioned access barriers. It cannot capture why one specific woman stayed silent for twenty minutes before naming hers”.
There is an additional risk that deserves particular attention in the Latin American context: the structural bias of AI models. Large language models were trained predominantly on English-language texts, under conceptual categories that reflect a specific epistemology: a particular way of knowing. When these models are applied to the evaluation of programs in Indigenous communities, in rural settings across Central America, or among populations historically excluded from knowledge production, the risk of reproducing biases is not theoretical. It is methodologically real.
An AI system that does not understand the logic of reciprocity in Andean communities might classify a practice of communal exchange as a barrier to implementation, when it is, in reality, a form of social cohesion that the program should have recognized from the design stage. A model without sensitivity to the political context of certain silences may generate interpretations that are technically consistent and culturally wrong.
The most recent literature on AI and Indigenous peoples makes this point clearly: cognitive imperialism: the imposition of categories of knowledge from the outside: finds in AI an especially efficient mechanism of reproduction, precisely because it operates with the appearance of objectivity (Perera et al., 2025). An algorithm has no visible ideology. But it was built by people with specific perspectives, in specific contexts, with data that reflects specific hierarchies of what counts as valid knowledge.
This critique does not imply rejection. It implies precision about roles. There are tasks in which artificial intelligence can genuinely contribute to strengthening participatory evaluation: not replacing it. The automatic transcription of interviews frees up time previously devoted to mechanical work, time that can now be directed toward analysis. The synthesis of extensive documentary reviews allows teams to contextualize their findings with greater rigor. The identification of thematic patterns across large volumes of qualitative data can be a valuable starting point, as long as it is not confused with the analysis itself.
The key lies in sequence and epistemic hierarchy: AI as a tool that expands the capacity of the evaluator, not as an arbiter of what the evaluation must conclude. Evaluative judgment: contextualized interpretation, the weighing of contradictory evidence, the construction of recommendations that are both politically viable and culturally relevant: remains irreducibly human.
And participation: the creation of conditions for those who live an intervention to name their experience in their own categories and on their own terms: is not a stage of the evaluative process that can be automated. It is its ethical foundation.
“Trust cannot be scaled. Listening cannot be optimized. Presence cannot be automated.”
Latin America has its own evaluative tradition, rooted in participatory action-research (Fals Borda, 1987) and in Indigenous epistemologies. It is a tradition that provides us with conceptual and methodological resources to navigate this moment with greater clarity than the uncritical adoption of frameworks produced in the Global North allows. From that tradition, I propose three orientations.
First, adopt AI on our own terms. Do not wait for ethical frameworks to arrive from UNEG or the OECD before deciding how to use it. Build, from Latin American evaluative practice, standards of responsible use that incorporate the perspective of the peoples and communities who are subjects of evaluations. Indigenous data sovereignty is not a technical matter: it is a political matter that must be at the center of any conversation about AI in evaluation.
Second, resist the pressure of efficiency when it comes at the expense of genuine participation. Donors and institutional clients will request faster, cheaper evaluations. AI makes it technically possible to produce them. But our responsibility as evaluators is to be explicit about what is lost when the speed of processing replaces the necessary slowness of participatory listening. That honesty is not always comfortable. It is, however, part of rigor.
Third, position ourselves as producers of evaluative knowledge, not merely as users of tools. AI models improve with the data that feeds them. If the Latin American evaluation community remains passive in the face of the development of these tools, models will continue to be trained predominantly with data that does not represent our reality. Participating in international debates on ethical AI in evaluation: within UNEG, EvalPartners, and the EvalIndigenous network: is a concrete way to influence that direction.
I return to that afternoon of silence and chocolate. To what is in no database. That woman, at the end of our conversation, told me something I was able to include in the evaluation report: though I had to contextualize it for it to make sense. She said, in her own words, that the program had come to her community talking about solutions, but without understanding that what they were living was not simply a lack of solutions. It was a lack of land. It was a lack of clean water. It was structural gaps that impact indicators captured only vaguely.
That is not data that artificial intelligence can produce. It is knowledge that emerges from relationship, from trust built with time and presence, from the willingness to listen to what the evaluation design did not anticipate. Participatory evaluation, at its best, creates the conditions for that knowledge to emerge and be recognized as legitimate evidence. Artificial intelligence, at its best, can help us process and communicate the evidence we already have. But it cannot substitute the process through which that evidence is produced.
That distinction is not merely methodological. It is ethical. As long as we maintain it with clarity, we will have something no algorithm can replace: the judgment to know when the data is not enough: and when what evaluation needs is, simply and profoundly, to listen.
Notes and References
OECD (2025). Aid at a glance: Development co-operation trends. OECD Publishing.
UNEG (2025). Ethical Principles for Harnessing AI in United Nations Evaluations. Document approved at the UNEG 2025 General Assembly. Available at: https://uneg.org/harnessing-ai
Perera, M. et al. (2025). Indigenous peoples and artificial intelligence: A systematic review and future directions. Big Data & Society.
Fals Borda, O. (1987). The application of participatory action-research in Latin America. International Sociology, 2(4).
