Reflecting on a research methodology involving beneficiaries in participatory evaluation

by Kylie L. Kingston

My participatory evaluation research journey was sparked by a motivation to involve young children, as the beneficiaries of an educational program, in evaluating their pre-school program. I wanted children to have the opportunity to participate in designing and using evaluation processes, so as to have their voice heard and to impact on programs and activities they were involved in.

Subsequently, this motivation broadened to include beneficiaries of any age and led to my masters and PhD research in which I sought to understand beneficiaries’ perspectives on participating in evaluating nonprofit organisations. After all, if beneficiaries don’t want to be involved in evaluation processes, requiring them to be is unlikely to be a useful, empowering or voice-giving experience.

Findings from my masters research indicated that beneficiaries indeed do want to be involved in evaluating programs provided by nonprofit organisation that they access. Of importance though is matching the way beneficiaries participate within the organisation (in relation to the programs they use and their timeframe of engagement) to the evaluation approach and processes. Further details about these findings and frameworks for participatory evaluation can be found in these two journal publications that stemmed from this research – Kingston, Furneaux, de Zwaan, and Alderman (2020a & 2020b). Extending these findings, my current PhD research seeks to further understand participative evaluation from the beneficiaries’ perspective.

In total, I have interviewed 31 beneficiaries (and additional employees and board members) across four different Australian nonprofit organisations. These organisations, ranging from small to large sizes, include those that provide community services such as health and rehabilitation support, housing advice and relief from social isolation, as well as professional membership support. When I reflected upon these interviews as a whole, the significance of talking with beneficiary stakeholders became apparent. Often research is done on beneficiaries and evaluation practices are developed for beneficiaries to use, rather than being developed with beneficiaries.

Along with co-authors, I have written about the importance of researching with beneficiaries in a recently published journal article titled “A Reflection on Critical Methodology: Accountability and beneficiary participative evaluation in Third Sector Research” in Voluntas: International Journal of Voluntary and Nonprofit Organizations. In that article, which focusses on the methodology used in my masters research, we discuss how the inclusion of beneficiaries in empirical research enabled their voices to be heard firstly in the research itself and secondly through the evaluation processes developed as a result of the research. We discuss how beneficiaries’ voices heard through the research process informed the development of participatory evaluation processes to be used within the nonprofit organisations ongoingly.

Within the Voluntas article, we identify three levels that researchers can consider when designing methodology for research projects that aim to empower and give voice to beneficiaries: the conceptual, processual, and reflexive levels. The conceptual level considers the ontological and epistemological assumptions that underpin the research, while the processual level considers the research methods, which in our example are case studies involving interviews with beneficiaries. We highlight here how interviewing beneficiaries became a means of representing this group within nonprofit research. “Finally, the reflexive level explores how findings from the processual level enable praxis through the development of approaches supporting beneficiaries’ participation in organisational evaluation processes. As such…research can engage beneficiary participation, in order to promote more effective beneficiary participation organisationally” (Kingston, Luke, Furneaux, & Alderman, 2021, p. 1).

My research motivation was to understand beneficiaries’ perspectives on participatory evaluation, and based upon those perspectives, to develop processes for their involvement. In this way beneficiaries can continue being heard through participatory evaluation processes. I hope the Voluntas article makes clear that beneficiaries’ involvement in research on participatory evaluation has supported their engagement with evaluation at an organisational level. The three levels of methodological design (conceptual, processual and reflexive) may be helpful for others designing research who seek to include marginal stakeholder groups, such as beneficiaries of nonprofit organisations, and enable them to have a stronger evaluative voice.