In Latin America and the Caribbean, the systematisation of experiences approach is widely known[1]. In some fields, such as popular education, it is even better known than evaluation. We ask ourselves, what differences and similarities are there between evaluation and systematisation of experiences? Is it possible to identify common aspects between systematisation and some types of evaluation?

At EvalParticipativa we wish to open the debate and for that we have invited Oscar Jara Holliday, one of the most renowned figures in the field, to kick it off.

The Context

According to specialised literature, systematisation of experiences aims at establishing learning as an essential element of any intervention policy. And he does so by rising to the challenge of promoting, designing and conducting learning processes in experiences that were probably not conceived with that purpose in mind. But is the search for learning exclusive to systematisation? Are there similar purposes in the field of evaluation?

The methodology of systematisation of experiences has unintentionally maintained a weak link with evaluation theory. In spite of various reflections that have attempted to formalise the method (Jara, 2018; Franke and Morgan; Chávez Tafur, 2006; Tapella and Rodríguez Bilella, 2014), dialogue and interaction between systematisation and other evaluation approaches have been significantly limited. At the same time, and probably due to its practical origins, this approach has not always made explicit its links to social theory, except for the connection that many authors have established with dialectical materialism (Capó S. et al, 2010; Ghiso, 1998).

We believe that the efforts to distinguish systematisation of experiences and evaluation should not lead to considering them as mutually exclusive approaches or lacking points in common. In this regard, Carden and Alkin (2012) provide a valuable resource in the form of their critical analysis of the second edition of Evaluation Roots (Alkin, 2012). In this analysis of the book they classify a series of evaluation approaches present in the context of low and middle income countries: (1) adopted methodologies (logical framework analysis, results based management, impact evaluation based on experimental and quasi-experimental methods), (2) adapted methods (participatory evaluation, developmental evaluation, outcome mapping, most significant change) and (3) indigenous methodologies. The latter are the methods that originated in the context of the global South, generally as result of collective rather than individual efforts and, given their practical origin, not based on a full prescriptive theory. In the case of Latin America, the method presented by Carden and Alkin is systematisation of experiences, which is added as another leaf to the Evaluation Theory Tree.

Well then, in this context, what does Oscar Jara tell us in this regard? We would like to invite you to read the note and to leave a comment with your point of view. Thank you!

SYSTEMATISATION OF EXPERIENCES AND EVALUATION: SIMILARITIES AND DIFFERENCES

By Oscar Jara Holliday ^[2]

Let us take as starting point the fact that both evaluation and systematisation are susceptible of being perceived as having different approaches and conceptions. Therefore, we should not worry about having close and unique definitions of them but instead aspire to find some fundamental guiding criteria, particularly with a focus on being able to apply said criteria to our concrete work and not wear ourselves out in the attempt to find abstract formulations.

The first thing we can affirm is that in general, in the fields of education, organization, community outreach and development, we work with a basis on projects that, depending on how we approach them, formulate aims, strategic objectives, both general and specific; goals to be achieved, as well as results, effects and impacts that are expected; they define components, activity plans, people in charge of realising them, indicators, schedules; sources and means of verification, products, necessary resources, potential risks, budget; mechanisms of monitoring, project follow-up and evaluation, etc. These projects are normally structured around a specific planning logic based on a diagnosis or preliminary study, a formulation of the institutional mission and vision, and other elements. This logic and its instruments are usually a fundamental reference for the evaluations that rely on them.

But from the very moment that a project is set in motion, a new component arises: a process. This process depends primarily on how the people involved in the execution of the project interpret it and feel about it as well as on how they act and how they interact among one another. This process, then, emerges as a vital component of the project and, without a doubt, unexpected elements that could not have been anticipated or planned ahead will appear in its course; therein lie its richness and importance.

The framework of the project will continue to be a reference point but it is the process in motion what will dictate the specific dynamic and direction: both hindering and facilitating factors will appear, some of which will come from elements external to the project (context, specific situations in which it is placed, the dynamics of relations among participants, etc.) while other factors will come from the project’s own dynamic. There will appear synergies that will boost and speed up the actions beyond initial expectations but there will also come up hurdles along the way that will hinder the progress. The project’s logic is always more lineal and prescriptive; the process’s is more complex, dynamic and unpredictable.

That is why we affirm as main point, that there must be a dynamic and synergistic relation between project and process (that is to say a relation in which each element affects the other in such a way that they result in an effect greater than the sum of their individual effects).

From this central point we can better understand exactly how evaluation and systematisation of experiences contribute as factors of learning through practice.

• Evaluation is closely related to the project and its completion. Systematisation of experiences is more related to the process, its dynamic, its path and its vitality: a view from and on the lived experience.

• Evaluation (regardless of type and approach) will always entail a value judgement, whereas systematisation’s objective is to recover the practices and learning generated within it so as to acknowledge the views that are generated from the perspective of the various stakeholders without necessarily passing judgement.

• In order to express a value judgement, evaluation requires a contrast to be established, within an institutional framework, between what was expected to achieve with a project and what was ultimately achieved. Systematisation can incorporate other dimensions that came up during the process and which might have no further relation to the institutional proposal that boosted the project.

• The majority of evaluations are carried out with the aim of providing information for decision making. This can sometimes result in them being primarily administrative, immersed in an approach focused on control and supervision rather than on the production of learning, especially when they are developed as external evaluations. This in turn can bring on attitudes that make some subjects get on the defensive for fear of an upcoming judgement and its implications.

• Systematisation of experiences tends to be freer of those administrative restraints and allows people to approach its practice with a more critical, self-critical and reflexive attitude, and willing to learn from what took place in the experience.

• The pace can also be very different. Many evaluations are very limited by time due to the need to present reports and make decisions based on pre-established periods. Systematisation of experiences, with a perspective more oriented towards the problematisation and the understanding of the stakeholders’ lived experiences by recovering their knowledge and their views, can be slower and may not be subject to such strict progress monitoring along the way. Rather, it builds a cumulative organizational learning beyond the projects’ periods.

• Due to its origin and some of its characteristics related to its use as a component for validating projects funded by governmental and non-governmental bodies alike, evaluation has developed into a professional specialization field. So much so that there are professional evaluators and even national and regional associations of these specialists. On the other hand, systematisation has emerged rather as a component integrated to the processes promoted by the stakeholders and although there are many people who do consulting work in this field, none of them would consider themselves a professional systematiser.

• Evaluation allows the gathering of vital information about the results, something which systematisation generally does not provide. In turn, this information and the balance supplied by evaluation are crucial for the reorientation of the projects and future actions. It is also essential to form value judgements about the goals, results, effects and impacts actually obtained, as well as on the reasons behind its achievements and its failures in order to correct or reaffirm what should be done. This marks the importance of evaluation.

Bearing all this in mind, we need evaluation as much as we do systematisation of experiences. And rather than worry about the abstract differences between them, we should focus on how to create, in the concrete practice of our projects and processes, the conditions for a fruitful, complementary and synergic encounter between these two exercises for producing knowledge from practice; an encounter that allows us to turn them into educational facts and critical learning factors with which to strengthen our skills in strategic projection and practice improvement.

…

[1] Translated into English, systematisation, which has its origins in Latin America in the 1960s, loosely means the act of organising something according to a system or a rationale. Actually, systematization involves not only linking somehow the activities with the products or outcomes of a project, but mainly accounting for why this relationship exists, aiming at unraveling the underlying connections, in order to understand the development of the experience, and how the change has (or has not) happened. We use the term “systematisation of experiences” as process of reflection and critical interpretation of lived experiences to generate meaningful learning from practice. This goes beyond “systematising information”, which is limited to organising, classifying and cataloguing data.

^[2] Popular Educator and Sociologist. Director of the Alforja CEP (Centre for Studies and Publications) in Costa Rica.

President of the Popular Education Council of Latin America and the Caribbean, CEAAL.

 

References:

Alkin, MC (ed.) (2012) Evaluation Roots, 2nd edn. Thousand Oaks, CA: SAGE.

Capó S, W.A., Arteaga C, B.A., Capó S., M.Y. (2010) La Sistematización de Experiencias: un método para impulsar procesos emancipadores. Venezuela, CEPEP.

Carden, F. and Alkin, M.C. (2012) Evaluation roots: an international perspective. Journal of MultiDisciplinary Evaluation 8(17): 102–18.

Chávez-Tafur, J. (2006) Aprender de la experiencia. Una metodología para la sistematización. Lima, Peru: Fundación ILEIA / Asociación ETC Andes.

Francke, M. and Morgan, M.L. (1995) La sistematización: apuesta por la generación de conocimientos a partir de las experiencias de promoción. Lima, Perú: ESCUELA para el Desarrollo.

Ghiso, A. (1998) De la práctica singular al diálogo con lo plural. Aproximaciones a otros tránsitos y sentidos de la sistematización en épocas de globalización. Mimeo. Medellín, Colombia: FUNLAM.

Jara, O. (2018) La Sistematización de Experiencias, práctica y teoría para otros mundos posibles, Cinde. Bogotá.

Tapella, E. y P. Rodríguez Bilella (2014) Shared learning and participatory evaluation. The sistematización approach to assess development interventions, Evaluation, vol. 20 no. 1 115-133, SAGE publications.