INCORPORATING PARTICIPATION IN PERFORMANCE EVALUATION

Lessons from a Mexican programme providing financial assistance for individuals with permanent disabilities

by Karen Azucena Hernández González

The ideas I share in this post are largely based on my experience working as an analyst and consultant in the 2018 evaluation of the Mexican Pensión para el Bienestar de las Personas con Discapacidad Permanente (Financial assistance to promote the wellbeing of individuals with permanent disabilities; hereafter PBPDP as per its abbreviation in Spanish) during its first year of implementation. This social programme is aimed at people with permanent disabilities aged between 0 and 29 years; Afro-Mexicans with permanent disabilities aged between 30 and 64; and people with permanent disabilities that live in severely marginalised areas (ROP, 2021).

As a young evaluator in the evaluation team, I had the opportunity to familiarise myself with the programme, conduct fieldwork and play a role in its respective analysis. While there was no explicit request in the project’s terms of reference to include a gender, intercultural or intersectional perspective, I was able to identify spaces where the issue should be explored in depth and where we could study ways in which the programme could contribute to finding solutions.

A singular performance evaluation

Our evaluation of the PBPDP programme was an external performance evaluation which explored the design of the programme and some of its operational processes. Performance evaluation takes into account decisions made during the planning process and then analyses the management and achievement of certain results. Our evaluation of the PBPDP adopted a qualitative methodology and combined an exploratory look at the design of the programme with fieldwork to analyse its implementation. The evaluation was carried out in response to a request from el Consejo Nacional de Evaluación de la Política de Desarrollo Social (Mexican Council for the Evaluation of Social Development Policy, hereafter CONEVAL).[2] The cross-cutting approach helped the programme identify areas for improvement and implement them in the 2020 documentation.

Returning to my experience, I identified that one of the critical moments in the programme’s design was the diagnostic stage. Furthermore, this was where most weaknesses were registered, due to the biases present during the information gathering stage and the programme diagnosis. When an evaluation addresses a complex social development issue and aims to close inequality gaps, it has to incorporate a disability, gender, intersectional, intercultural and human rights approach in order to understand the context and performance of the intervention.

A case in point

During the fieldwork I met Luis, aged 60. From his behaviour and the background information we were given, it seemed he had a psycho-social disability and difficulties in communicating and fulfilling his potential within the community. Luis lives in a rural area categorised as extremely poor (CONEVAL 2020), in the municipality of Cuetzalan, Puebla. He is cared for by his brother and sister-in-law. They did not know exactly what ‘illness’ he had, as he had never been seen in a health centre; they only knew that he was unwell.

This example highlights a lack of awareness, cultural barriers and insufficient access to healthcare and wellbeing assistance for people with disabilities. This is closely related to issues of deprivation and marginalisation which place people like this in vulnerable situations. It also exacerbates the gaps of inequality as they barely have access to basic health, education and care services, coupled with stigmatisation which keeps them in socially marginalised situations. The case of Luis is therefore a clear example of the complex circumstances in which the target population lives.

If we apply a disability perspective, we can explore the intersectional characteristics of people with disabilities and consider the type of disability, gender, age, socio-economic status, place of residence, indigenous identity etc. Considering these intersections is important when seeking to protect people’s individual and collective rights. It is particularly pertinent when the programme under evaluation targets diverse age groups, women and men, people of diverse sexual orientations, those living with disabilities, indigenous people living in highly marginalised areas, speakers of indigenous languages and Afro-Mexican people.

Therefore, when approaching and theorising the issue and its impacts on the lives of people with disabilities, it was important to integrate a participatory approach intended to engage and involve the ‘people who are (directly or indirectly) related to the project or evaluation’ (Mideplan, Focelac, Conapdis, 2019) who should, of course, contribute their points of view according to their lived experience.

This approach should also be intercultural ‘in order to guarantee recognition, sensitivity and respect for existing cultural differences, according to the concept that cultures can be different but are equally valid, and that there are no superior or inferior cultures’ (CUSCO, Defensoría del Pueblo, Red de Género e Interculturalidad, n.d.). With these ideas in mind, the evaluation was designed to ensure we heard the voices of people with disabilities, service providers, programme implementers, decision-makers, and carers. Understanding the issue from a gender and intercultural perspective is key as ‘the cultural dimension of the problem […] forces us to look at the detail, the context, the power relations, the patriarchal system of social organisation […] and to look for culturally situated evidence’ (FIMI, 2013:7).

In my opinion, although the programme made passing references to exploring these differences, they had certainly not been fully considered in the intervention’s methodology, and much less in the creation of participation spaces. The evaluation identified the main areas for improvement to be related to the construction and understanding of the problem, starting with the need to clearly define what is meant by ‘permanent disability’ and ‘obvious disability’.

Furthermore, we identified the need to develop a proper evidence-based description of the target population, given that the programme implementers did not know how to clearly define who had an obvious disability, confusing incapacities with disabilities. There was also no disaggregated information available to identify which of the people with disabilities were of Afro-Mexican or indigenous descent, so it was not possible to measure inequality gaps or other aspects in the evaluation of the fieldwork design requested by CONEVAL.

Lessons

Finally, I think evaluators can benefit from incorporating the participation of key social actors in order to: (1) recognise people’s everyday strategies for adapting to the world; and (2) incorporate these strategies into the problem’s solutions. This allows us to be clearer about whether an intervention is fulfilling its purpose and the extent to which it is impacting people’s lives. The living conditions of people with disabilities are particularly complex in terms of social interaction and this represents a major challenge that should be taken into account when weighing up the risks and methodological needs involved. In this case, we found that most carers of people with disabilities are women, so adopting a gender perspective is also central to future evaluations.

REFERENCEs

Leave a Reply