by Pablo Rodríguez Bilella
Over the past two decades, the movement aimed at achieving evidence-based policies has gained importance and prominence. It holds that policymakers should base their decisions on the best available evidence regarding “what works,” rather than on ideologies or in response to particular interests.
One of its assumptions is that not all evidence has been rigorous enough to provide certainty in decision-making. This accentuated the orientation toward supporting certain approaches and methodologies that, given their rigorous formulation, would lead to superior results. This general context has favored debate and discussion on impact evaluation from different spheres and spaces (political, academic, social, etc.).
Thus, the increase in the number of impact evaluations over the past fifteen years has been notable, accompanied by reflection (and concern) about the quality of their methods, conclusions, and recommendations. Their growing relevance corresponds to the fact that knowing the impact of a particular intervention (a program, a project) facilitates decision making regarding its continuation, its modification, or its termination.
Whichever course is taken, impact evaluation should facilitate learning regarding how to replicate or scale up a pilot experience, even providing criteria for its adaptation to another context. Alongside this learning component is the accountability component of the intervention, both for those who finance it (donors, public resources, etc.) in order to show that resources are being prudently invested, as well as for different social actors (communities, beneficiaries, civil society organizations).
The monitoring and evaluation work of projects, programs, or public policies has long tended to focus on reporting certain processes or on the use of resources (are we implementing the program within the planned timeframe, with the intended costs?), as well as on what was done or carried out (the so-called outputs: how many workshops were delivered? how many people participated?). However, the central point in impact evaluation is to find out whether the situation that gave rise to the intervention has changed, and whether the intervention had anything to do with that change. In other words, whether or not the change in the problematic situation can be attributed to the intervention, and to what extent.
For some lines or schools of impact evaluation, the question of attribution is resolved by seeking to answer the question: what would have happened if the intervention had not taken place? This goes beyond an analysis of the situation before and after the intervention. These questions justify the emergence of the concept of the counterfactual or counterfactual scenario: what would have happened with the situation of intervention if the program or project had not been developed?
In academic and political circles, there has been an intense debate about what should be considered a rigorous impact evaluation. This debate has tended to polarize between those who advocate the application of experimental and quasi-experimental methods, versus those who advocate greater breadth and relevance in the use of other methodological approaches. The former have been referred to in the literature as “randomistas,” in reference to their adherence to the methodology of randomized control trials (RCTs), arguing that randomization is the only means capable of ensuring that unobservable selection bias is taken into account. That is to say, the selection of who will be beneficiaries of a given intervention versus the selection of who will form a control group must be done at random and starting from a common universe of actors: everyone should have the same chances of ending up in one group or the other.
On the opposite side of the randomistas are those who argue that randomization is exceptionally appropriate to be used in the evaluation of development interventions, suggesting that barely 5% of development programs are feasible to be evaluated using an RCT-type design. And even when the use of such a design is indeed appropriate, the use of counterfactuals only allows answers to contingent questions associated with particular contexts, and their findings cannot be used to generalize to other scenarios, unless they are accompanied by more detailed knowledge of the causal mechanisms operating in the process that goes from the possible cause to the effect.
In the publication Diseños y Métodos para la Evaluación de Impacto (only in Spanish), we hold that impact evaluation includes any type of evaluation that systematically and empirically investigates the impacts that an intervention produces. This broad view of impact evaluation allows accounting for the interest in long-term emphasis and the complex nature of development interventions.
Impacts have generally been defined as those results achieved by an intervention in the long term, however many of the current examples of impact evaluation are pointing to intermediate or short-term results. At the same time, the complex nature is recognized through the space given to uncertainty and the emergence of unexpected (and undesired) effects of development interventions. In this sense, and in the interest of recognizing such complexity, there is a growing interest in recognizing and understanding how particular interventions “contribute” to achieving a certain impact, rather than focusing only on issues of “attribution.”
This is particularly visible in the case of complex programs and long-term interventions focused on issues of governance, democracy, and accountability, where estimates of “net effects” become almost impossible to address, reducing the emphasis on “measuring impact.” On the other hand, the emphasis on the systematic and empirical approach leaves the field open to a plurality of designs and methods, without reducing impact evaluation to the application of a single design or method in particular.
After introducing a reflection on the questions that every impact evaluation seeks to answer, as well as the actors capable of carrying it out, the publication “Diseños y Métodos para la Evaluación de Impacto”, develops six key aspects to consider in the choice of methods for this type of evaluation: clarifying values, generating a theory of change, measuring or describing impacts and other variables, explaining whether the intervention was the cause of the observed changes, synthesizing evidence, and reporting findings and supporting their use.
Finally, if we recognize that there are alternative methods for carrying out an impact evaluation, it is possible to consider that the most appropriate designs must be deliberately chosen in each case. The current context of impact evaluation development accounts for new paradigms of causal analysis that reject the simple distinction between quantitative and qualitative methods, recognizing the possibility of combining different methods, both quantitative and qualitative, as well as different qualitative methods.
In addition to the generally more developed approaches, three design approaches are not widely implemented in impact evaluation and offer considerable potential for linking development interventions with outcomes and impacts. Given their usual absence in training on impact evaluation, and recognizing their merits, capacities, and promise to reinforce and strengthen the current practice of this type of evaluation, the publication concludes by making a particular reference to (i) theory-based approaches, (ii) case studies, and (iii) participatory approaches.
