by Ricardo Ramírez & Joaquín Navas
‘Readiness’ is an English language term that is difficult to translate into Spanish as it encompasses various themes: preparation, availability, propensity, timeliness, agility, competency, willingness – among others. In the Spanish version of this entry, we use ‘availability’ (‘disposición’) with the understanding that it covers the other themes. In the field of evaluation, readiness is foremost across the first two steps of Utilization-focused Evaluation (UFE) as proposed by Michael Quinn Patton (2008). Readiness is also referenced in other fields, for instance in communication strategy design (Ramelan, 2014).
While the goal of UFE is the actual use of evaluation, the approach relies on the direct participation of ‘evaluation users’; it is their direct engagement that provides them with a sense of ownership over the process and the findings. In other words, UFE confirm that it is very much part of other collaborative approaches to evaluation (Ramírez & Brodhead, 2020).
In conventional evaluation, readiness is relegated to the procurement of an external evaluator, with the control over the evaluation design in the hands of the funder or of the external evaluator. In contrast, under UFE, readiness of the program and of the evaluator constitute a foundational step before the evaluation can begin. This is followed by the identification of the ‘primary interested users’ as the persons who will be responsible for the design of the evaluation, and in charge of using the findings and following up on the recommendations (Ramírez et al., 2018).
PROMPTS FOR EVALUATORS
The following are three prompts about readiness that have emerged from practice; they are prompts to guide practitioners:
-
- The power to design
- The commitment to learn, and
- The emphasis on building an evaluation culture (Ramírez et al., 2018).
The power to design addresses the potential got a project or organizational team to participate in the design of an evaluation. When it comes to defining the purpose and scope of the evaluation, this depends on the extent to which a funder is willing to share decision-making power with a grantee. This can happen when the funder perceives some value added when the grantee (partner or beneficiary organization) co-designs the evaluation as a learning tool. There are precedents where the funder and grantee co-designed evaluations (New Economy Development Group, 2014; Ramírez et al., 2017), and several others where the funder entirely delegated the design to the partner (Ramírez & Brodhead, 2013).
While it is tempting to assume that it is common for project implementers to be able to take ownership and learn through the co-design of evaluation, this is certainly not the norm. It is quite standard even for project implements to favour having an external evaluator take on evaluation design responsibility. This may be for various reasons: lack of time, a rush to produce reports, credibility in the eyes of third parties, etc. It is therefore paramount during the readiness assessment to confirm the extent to which there is a willingness and commitment by the implementing organization -let along the funder- to participate in the design of the evaluation.
The commitment to learn through evaluation becomes palpable when the primary users -with the facilitation by the UFE evaluators- propose evaluation ‘uses’ or ‘purposes’ that are complementary to conventional accountability ones. Such uses open the door to learning about the ‘why’ and the ‘how’ of a project; they open the door to discovering the factors and procedures that allowed a project to reach its goals, as well as the barriers encountered. This process invites the primary users to pose questions on various themes, such as: leadership, stakeholder engagement, on the outcomes of communication strategies, on adaptive management, among others.
In order to cement an internal UFE capacity within a project or organization, it is necessary to discover or enhance an evaluative culture. This attribute depends to a large extent on the leadership of a project or organization. Funders who are committed to UFE will need to seek out partners with such characteristics. For instance, during the implementation of a UFE it is common to witness the use of preliminary evidence before a final report is ready. Such instances demonstrate that adaptive management is part of the organizational culture, one that can be nurtured with evaluation. The table below provides a summary of the various readiness themes, across three dimensions: the program or project, the evaluator, and the funder; as proposed by Zaveri et al. (2017).
In summary, readiness requires attention from the start by the project or organization, the evaluator, and the funder. If the level of readiness is not high, the potential to introduce collaborative approaches to evaluation like UFE will be limited, especially when the funder takes the role of single evaluation user, and with a focus solely on measuring impact.
In practice, it is common to detect a high level of readiness early on among a project team for UFE, only to witness its erosion as time goes by due to other work pressures. It is for this reason that readiness demands constant nurturing. For evaluators committed to collaborative approaches, readiness requires attention from the moment one reviews a request for proposals.
tools
The following tools are useful to address readiness:
UFE Checklist. In Patton’s UFE 4th. edition book (2008), there is a checklist for each step, with readiness of the program and of the evaluator as the first two.
UFE competency self-assessment. Based on the five competency domains by the Canadian Evaluation Society, this self-assessment tool helps evaluators verify their level of expertise (DECI-4 Project, 2022a y b).
Organizational readiness self-assessment. This self-assessment tool enables a project or organization to review the different dimensions of readiness as a basis for interacting with the evaluator. There are elements that are difficult to change (such as an organizational culture, or a limited power to influence the terms of reference of an evaluation), while others can be modified (more senior management buy-in, or the allocation of resources.
REFERENCES AND RESOURCES
DECI-4 Project (2022a). Utilization-focused evaluation self-assessment tool.
DECI-4 Project (2022b). Competency summary: Findings and analysis of the self-assessment feedback by CPC Partners.
New Economy Development Group. (2014). The evaluation of Research ICT-Africa: A case study.
Patton, M.Q. (2008) Utilization-focused evaluation 4th. Ed. Sage.
Ramelan, V. (2014). Research communication: concepts, steps, readiness, purposes and audiences. Webinar (21.5 minutes).
Ramírez, R. & Brodhead. (2013). Las evaluaciones orientadas al uso: Guía para Evaluadores. Penang: Southbound.
Ramírez, R. & Brodhead. (2013). Utilization-focused evaluation: A primer for evaluators. Penang: Southbound.
Ramírez, R. & Brodhead. (2014). Readiness & mentoring: Two touchstones for capacity development in evaluation. Paper presented at the CDI Conference: Improving the use of M&E and findings. Wageningen, 20-21- March.
Ramírez, R., Kora, G., & Brodhead, D. 2017. Translating project achievements into strategic plans: A case study in Utilization-Focused Evaluation. Journal of MultiDisciplinary Evaluation 13(28): 1-23.
Ramírez, R. & Brodhead, D. 2020. Comparing the validity of two sets of evaluation principles: Adding value to both. pp. 185-211. In: Cousins, B. (Ed). Collaborative Approaches to Evaluation. Sage.
Ramírez, R. & Brodhead. (2021). Readiness: The key to positioning and achieving utilization-focused evaluation. Expert tutorial presented at the Canadian Evaluation Society Conference (on-line).
Ramírez, R.; Brodhead, D. & Quarry, W. (2018). Readiness in evaluation: Three prompts for evaluators. Canadian Journal of Program Evaluation 33(2): 258-267. doi: 10.3138/cjpe.42238
Zaveri, S.; Ramírez, R. & Brodhead, D. (2017). Confirming readiness for collaborative evaluation. Paper presented at the American Evaluation Association Conference, Washington, DC. 10 November.
