We are still receiving reflections and lessons from the invitation we launched a few days ago. This time, Olga Nirenberg, a colleague EvalParticipativa member through CEADEL, shares some of her thoughts with us on a participatory evaluation experience evaluating the health services of a Latin American NGO.

Remember that everyone can participate with their thoughts and lessons!

Prologue

Here I’m going to share an experience that at the time I found negative but which, looking back now, I think was a good lesson for me, the team and especially the women who participated in this “evaluation saga”.

I think that evaluation often clearly portrays differences in stakeholder perspectives and the importance of having their interests and influences mapped out. Furthermore, all stakeholders should always be fully involved. In other words, there should be a multi-stakeholder focus with clear decisions around which voices will be given priority.

The recommendation that arises from the experience that will be shared below touches on the ethical issues involved in participatory evaluation. This means that those of us who are evaluation professionals and who know about methodological issues, have the moral obligation to amplify the quietest voices and those that are not always heard, especially when it is claimed that these same people are the “recipients” of the initiative in question.

Another issue that is highlighted by this experience is that the people who approach us and contract our services, may think that they have privileges and control or can even manipulate the conclusions. This was the case of the NGO provincial leadership in question. Moreover, in my opinion it’s important to negotiate the terms of the report with those who contract the evaluation especially regarding the tone or ways of expressing certain elements. But this does not mean that you should leave out things that need saying.

This experience demonstrates some of the conflicts that can arise when those who contract us to facilitate evaluation think that they have absolute right to veto. It also demonstrates how powerful evaluation can be in bringing about changes.

Evaluation characteristics and context [1]

The evaluation took place (from 2010 to 2011) in two provincial capital cities in Northeast Argentina at the request of a Latin American NGO which aimed to improve access for women with low income to financial services, human development and primary health care.[2]

The NGO works in each province and there is Provincial Health Coordination and several Focal Points which guide its service users in areas related to the three components mentioned. Each Focal Point has several Community Health Contacts, each in charge of a group of female service users. The Community Health Contacts are specifically trained to guide the women in terms of accessing and using health services as well as sharing information and preventative measures. They are volunteers, also service users, chosen by their peers. They receive some incentives from the NGO and can be changed each year, although they can also be re-elected.

The evaluation focused on the Primary Health Care component. It explored the needs and demands, how services were used, as well as service user compliance, preferences and proposals. It also analysed the local availability of First Level of Care and Primary Health Care services. It triangulated qualitative and quantitative techniques and included participation from different stakeholders especially the NGO service users which includes the Community Health Contacts.

Preliminary visits were made to each province in order to contact the NGO’s provincial workers as well as health professionals and service user leaders (the Provincial Health Coordination for the NGO and the Community Health Contacts). We also requested pre-existing information (which turned out to be insufficient). In this way, we tried to understand as well as possible the contexts, establish basic agreements on the evaluative design, know which dimensions should be considered and what requirements were needed to make the fieldwork viable. This inroad was useful for mapping out the stakeholders in each province and was embellished during the fieldwork.

Main lessons

The stakeholder perceptions and explanations of the problems differ according to each stakeholder’s position in each specific context in addition to cultural, ethnic, generational and gender factors etc. For this reason, absolute consensus is  unrealistic for describing, explaining and prioritising problems as it is for proposals for how to overcome them.

The important thing in these evaluations is bringing together stakeholders, generating spaces for discussion, confrontation and joint reflection so that basic agreements can be reached – rather than consensus – for various aspects especially in the final stages including conclusions (reasoned evaluative judgements) and recommendations.

Furthermore, in the evaluations on access and quality of services, the service user perspective should be given priority, in this case, the NGO female service users. Although this was agreed at the beginning with the NGO provincial leaders, a conflict arose when they questioned the conclusions in the evaluative report and went on to question the participatory method that had been agreed, requesting an epidemiological study. Confronted with this stance, the Community Health Contacts requested intervention from the InterAmerican Development Bank, the NGO’s main funder.

From the perspective of the female service users, it was confirmed that evaluative initiatives work best when they combine quantitative and qualitative approaches, when they are multi-stakeholder and participatory and when they include moments of learning and empowerment, in this case something that they benefitted from themselves. Their testimonies affirmed that the different evaluative instances enabled them to:

• • learn from their peers;
  • reflect on the survey questions and the discussion guidelines;
  • learn more about their own situations and abilities;
  • channel their complaints more efficiently;
  • find new paths to continue advancing.

The evaluation enabled them to change the NGO’s provincial leadership and redirect the health care model so that it would meet the needs and preferences of the service users and be applicable within the local context due to the availability of services.

Ways of overcoming conflicts and obstacles

The first obstacle was the total lack of connection between the NGO and the Provincial Health Ministry. This was a problem due to the fact that they contracted both private and public health centres. This made it difficult for us to visit the public Primary Health Care Centres where the majority of service users traditionally preferred to go and who questioned the care given in these contracted centres. The evaluators managed to get the Provincial Health Ministry to authorise us to visit and observe the centres in order to evaluate the public Primary Health Care Centres too.

A second obstacle arose due to the fact that several agreements that had been reached to make the fieldwork possible were not enacted by the NGO. For example, finding appropriate places for group activities, making photocopies of material, buying flipchart paper and markers, inviting the service users to the meetings. In summary, little support was provided by the NGO leadership to make the fieldwork viable. This was attributed to the fact that the evaluation was more a requirement from the InterAmerican Development Bank rather than motivated by the NGO itself.

In order to overcome these obstacles, the fieldwork took double the amount of time expected. The conflict with the NGO’s provincial leadership was overcome with the intervention of the InterAmerican Development Bank which supported the evaluative report and promoted changes to the leadership and the model used for Primary Health Care services in the two cities.

Evaluation Team:  Dra. Olga Nirenberg (Leadership),  Lic. Marilú González de Ganem (Fieldwork coordinator),  Lic. Graciela Cardarelli (Systemisation of qualitative information),            Lic. Federico Sedano Acosta (IT support).