by Sergio Martinic
Participatory Evaluation experiences value and validate the interpretations of participants concerning the projects, policies and interventions that are being analysed (Fetterman, 2005). These experiences create space for participants to share their knowledge and particular ways of looking at the real-life situations and problems that the interventions under evaluation are seeking to address (Gil & Heras, 2010; Menéndez, Torralbo & Luque, 2021; Paño, Rébola, & Suarez, 2019).
This constitutes one of the principal contributions of this evaluative approach: knowing and understanding the actions from the point of view of the other allows for a deeper and more realistic examination of the results and impacts of the programme or action under evaluation.
The integration of participant perspectives brings with it significant conceptual and methodological challenges. This article argues that the opinions of participants should be analysed on the understanding that they are part of broader social representations of the culture to which the subjects belong. From a methodological point of view, qualitative data and discourse analysis strategies are good tools for analysing interpretations and knowledge expressed during a participatory process.
Levels of knowledge and interpretation analysis
The local knowledge, interpretations and insights expressed by participants constitute a gateway to a broader vision of reality, steeped in culture, that can be analysed with different levels of depth. In other words, they constitute the tip of the iceberg.
The first level is descriptive, and consists of accounting for and comparing the discourse of actors as they respond to the questions and issues raised. Usually, their opinions are presented and analysed textually using excerpts grouped into categories and presented in matrices or tables, to facilitate comparison.
The second level focuses on the associations or meanings evoked by the texts that have been collected. These meanings —which allude to the semantics of the texts— link the opinions to the traditions and culture from which they emerge. This level of analysis refers, among other things, to the distinctions, classifications and principles learned from everyday life, which subjects select and use to elaborate opinions and interpretations and to guide their actions in different situations. For example, critical views of sex education projects, or resistance to them, may be associated with the religious beliefs of their families. Likewise, certain technological innovations in Andean communities may impact beliefs and rituals associated with respect and care for Mother Earth (Pachamama).
These two levels are closely related. An individual’s discourse is always influenced by shared distinctions and principles that come into play when a particular opinion is produced. These principles shape peoples’ thinking and highlight the limitations and possibilities established by the culture to which they belong.
In any area of social life, be it work, health, political life, social organisation or daily interaction, individuals deploy ways of understanding and interpreting reality according to this framework and to the system of distinctions and classifications provided by the culture of which they form a part.
Thus, the challenge of participatory evaluation is not only to describe the thinking and local knowledge of the actors involved, but also to understand the underlying cultural principles and the particular ways these are associated with the texts and records that are produced. Ultimately, it is a question of understanding how individual opinions are produced within a broader perspective that is influenced by the history, experience and culture of the community they belong to.
Serge Moscovici (1979), a French social psychologist, introduced the concept of social representations to address issues associated with common sense thinking in everyday life. For Moscovici, representations constitute socially constructed practical knowledge acquired through common experiences, education and social interactions. This knowledge gives meaning to, and helps to interpret, shared facts and actions.
Social representations constitute reference systems that give a sense of logic and coherence to the world by structuring the different ways facts are explained and the relationships between these explanations. They are not a mere reflection of something external, but rather constructions that give meaning to, and make sense of, the object or reference point in question.
Social representations have three dimensions that are key to this discussion. Firstly, the informational dimension that informs the cognitive distinctions, concepts and terms used to interpret and make sense of the reality under examination. Secondly, the structural dimension, which provides ways of ordering and organising the ways in which the units or parts of any social representation relate to each other. Thirdly, the normative ethical dimension, which determines how valid, desirable or legitimate these distinctions and connections are. Subjects participate in the systems of ideas and will therefore value as positive or good whatever the representations in which they participate define or value (Moscovici, S, 1979).
Ultimately, social representations constitute socio-cognitive systems in which stereotypes, opinions, beliefs, knowledge, values and norms tend to invoke positive or negative attitudes. These representations act as a classificatory system and as the guiding principles governing the practices and interactions of the subjects. They constitute shared principles, codes and blueprints of thought, which regulate everyday interpretations and interactions by defining the limits and possibilities affecting the lives of particular groups and cultures (Araya, 2002; González Rey, 2008).
In a participatory evaluation, it is important to analyse the opinions of the participating actors within a broader framework that reflects the way they represent the problem or the strategy that is being implemented by projects and social interventions.
A range of different strategies exist for analysing qualitative data, all of which insist that this analysis does not consist simply of a “textual” and descriptive presentation of information. On the contrary, this kind of analysis seeks to illustrate the principles of distinction and of cultural grammar that organise and give meaning to the views of actors concerning the issues or problems raised.
These methodological strategies include Grounded Theory (Strauss, A. & Corbin, J. 1992) and Semantic Structure Analysis (see Rémy, 1991; Hiernaux, ). The two strategies follow similar steps: (a) data mining; (b) construction of classification categories or codes; (c) understanding the ways these categories relate to each other; and (d) creation of interpretive models, using central or selective categories. Grounded Theory emphasises the theory or narrative created by the subjects, while Semantic Structure Analysis focuses on the principles used to organise the analysed texts.
Semantic Structure Analysis is also useful for exploring the distinctions and classification systems that are expressed in the opinions of participants. The theoretical antecedents of this approach are found in structural semantics and, in particular, in the work of A.J. Greimas (1966), which has been used extensively in the analysis of representations and the symbolic models that influence the production of such discourses in the field of social sciences, particularly in the French tradition (Hiernaux, 1977, 2009). In Latin America, the method has been applied in a variety of cultural studies that include Martinic (2006) and Suarez (2008).
This method of discourse analysis is closely related to the question of representations raised by Moscovici. It is, precisely, a method that seeks to analyse the three dimensions mentioned above, characteristic of any representation, namely: the cognitive, structural or organisational and finally, ethical and normative dimensions. Once the object of the evaluation has been defined and the necessary material has been gathered, an initial descriptive analysis is carried out which focuses on the units of meaning that make up the texts or corpus under analysis (composition). The minimum units of meaning are the codes, constructed using the oppositional principle. In other words, concept “A” is opposed to or distinguished from concept “B”. Thus, the code reflects a relationship between concepts or categories, and accounts for classifications and distinctions that are explicit or implicit in the discourse of the subject (Martinic, 2006).
Once the codes and their categories have been defined, the next stage advanced by this method is to identify the relationships that exist between these units or categories (combination). This constitutes a second interpretive stage, intended to describe the ways the categories relate to each other in the context of the material under examination. During this stage, the opinions of different subjects are compared and those that are close or similar in meaning are incorporated into the same code. The analysis thus moves from individual distinctions or classifications to others that are shared by the group of actors under consideration.
The results of this analysis are presented in diagrams or structures that illustrate the ways the codes relate to each other. For example, they might be presented as parallel lists (e.g. identifying the positive or negative aspects of a project’s impact on the quality of learning in rural schools). Or they might adopt a more complex approach, presenting comparative results or double-entry tables: an approach that is more complex but which makes it possible to account for the nuances or variations that may exist in the distinctions expressed by the different members of a group. For example, positive and negative opinions may be held about the same topic, both of which are equally valid.
The codes constructed in this way, and the relationships established between them, enable systems of shared categories and classifications to be defined that are related to the knowledge, values and contents of the culture to which they belong. An understanding of these systems of distinctions recovers a particular way of classifying and ordering experiences, from the point of view of the participants. This highlights the limits that, in the form of local knowledge or principles of action, guide the interpretations and practical actions of subjects in relation to the real-life problems they face.
In summary, Participatory Evaluation presents an opportunity to gather participant knowledge and insights that are rooted in their cultures and identities. As we seek to better understand these perspectives, the proposal is to go beyond the textual recording of opinions. This is a first level of analysis that accounts for the individual opinions of subjects, in a context of participatory dialogue. But the important thing is to understand the meaning of the text, and its associations with shared meanings that are a part of the local culture’s traditions of thought and action. From the methodological point of view, strategies based on Grounded Theory and Semantic Structure Analysis constitute rigorous and validated procedures for the comparative study and interpretation of knowledge, insights and social representations of participants.
This contribution was presented at the session organized by EvalParticipativa for the Participatory Action Research and Evaluation Conference (PAREC) in April 2022.
Araya, S. (2002). Las representaciones sociales. Ejes teóricos para su discusión (Cuaderno de Ciencias Sociales No. 127). Cuaderno de Ciencias Sociales (Vol. 127). San Jose.
Fetterman, D. (2005). Empowerment Evaluation Principles in Practice. In D. Fettermn (Ed.), Empowerment Evaluation Principles in Practice (pp. 42–72). Guilford Publications. https://doi.org/10.1177/1090198105284876
Gil, E., & Heras, L. (2010). Evaluación participativa y empoderamiento: análisis documental de investigaciones y prácticas. pp 1–13.
González Rey, F. (2008). Subjetividad social, sujeto y representaciones sociales. Redalyc-Org, 4(2), 225–243.
Greimas, A.J. (1996) Sémantique structurale. Paris : Larousse.
Hiernaux, J.P.(1977) L’Institution culturelle II. Méthode de description structurale. Paris: PUF.
Hiernaux, J.P. (2009) El pensamiento binario. Aspectos semánticos, teóricos y empíricos. Cultura y Representaciones Sociales 3, (6) 25-42.
Martinic, S. (2006). El estudio de las representaciones y el Análisis Estructural de discurso. En: Canales, M. ed. (2006) Metodologías de investigación social. Santiago, Lom, pp. 299-320
Menéndez, S., Torralbo, M., & Luque, S. (2021). Guía práctica para la planificación y evaluación participativas de las políticas públicas. (S. Menéndez, M. Torralbo, & S. Luque, Eds.), La Participación transversal. Andalucía: Junta de Andalucía.
Moscovici, Serge (1979) El psicoanálisis, su imagen y su público. Buenos Aires: Huemul.
Paño, P., Rébola, R., & Suarez, M. (2019). Procesos y Metodologías Participativas: Reflexiones y experiencias para la transformación social. (P. Paño, R. Rébola, & M. Suarez, Eds.), CLACSO. Buenos Aires: CLACSO.
Strauss, A. & Corbin, J.(1992). Basics of qualitative research: grounded theory procedures and techniques. Sage, Newbury Park.
Suarez, H. (2008) El sentido y el método. Sociología de la cultura y análisis de contenido. México: El Colegio de Michoacán /Universidad Nacional Autónoma de México, 2008, 330 páginas.