EBA arranged a session on evaluation methods on October 19th, 2017, at the biannual conference of the Swedish Evaluation Society. Under the steer of Kim Forss, member of EBA, three authors presented their own take and discussed that of the other participants on how to evaluate the impact of aid. We were treated to a high level discussion of great interest to evaluation practitioners and researchers alike. In spite of the condensed format of the session it was sure to pique the interest also of those in the audience with less previous exposure to the approaches.
Although methods development is not a prime concern for EBA we have nevertheless commissioned three studies on methods/approaches that may be used to evaluate the effects also of aid interventions. The three approaches described so far are Randomized Control Trials (RCTs), Qualitative Comparative Analysis (QCA) and Geospatial Impact Evaluation (GIE). In commissioning these studies EBA wants to contribute to the discussion on adequate and appropriate methods for assessing aid given the large number of evaluations carried out in the aid sector. However, it is important to note that EBA is not advocating for any particular evaluation approach or method.
The authors of the three studies, Anders Olofsgård (EBA 2014:01), Barbara Befani (EBA 2016:05) and Ann-Sofie Isaksson (EBA 2017:09) presented briefly the main features, strengths and weaknesses, of their respective approach. Subsequently Barbara Befani compared the three methods followed by a joint discussion with the participants and the audience.
Anders Olofsgård argued for using randomized control trials (RCTs). Simply put this approach studies the effect of a treatment administered to a group as compared with a group not receiving the same treatment. Olofsgård considers this procedure to be superior in terms of interior validity although weak on exterior validity. Ethics may also be a consideration, as are cost and the time required. On the positive side Olofsgård listed avoidance of political capture and rigour as planning for the evaluation ideally should take place already at the design stage of an intervention.
Barbara Befani gave an overview of Qualitative Comparative Analysis, QCA, presented as a quick, simultaneous testing of multiple theories of change to extract common causes across different types of interventions. Barbara argued that it is not necessary to construct a counterfactual in order to establish causality and that there are at least three different approaches to this end. One of the advantages of QCA is that it allows for several causal factors to be viewed in combination as ”packages”. The method requires at least 5 comparable cases and input from experts on the substance of the cases as well as conceptual knowledge. An additional strength of the method is that it combines qualitative data with quantitative analysis. It is also relatively strong on external validity. A weakness of the approach may be internal validity but this should be possible to handle according to Barbara Befani.
A relatively new approach, Geospatial Impact Evaluation (GIE), was presented by Ann-Sofie Isaksson. The main feature of this evaluation practice is the use of data on the localisation of programmes and projects that are matched with relevant outcome data from surveys, satellite imagery, mobile phones, internet use etc and analysed with quasi-experimental methods. Studies have been made of local conflict, aid, corruption and environmental changes using this approach. The strength of this approach is that it lends itself to quasi-experimental analyses with relatively strong generalizability. In addition it is quite cost effective as it is possible to use existing data material. Another advantage is that GIE allows for analyses at an intermediate geographical level, between individual projects and the national level. Ann-Sofie Isaksson argued in favour of geocoding Swedish aid data as many actors already are doing this. Geocoded aid data are useful not least for recipient countries for analytical and monitoring purposes.
Barbara Befani then discussed the three approaches by pointing out similarities and differences and whether the three approaches might be combined.
Befani viewed GIE less as a particular method than an approach related to the particular data used. RTC and QCA are more methods in themselves. She argued that each of the three methods/approaches had advantages in responding to particular evaluation questions, given their characteristics. Geospatial Impact Evaluation is concerned with spatial data collection, RCTs are geared to controlling exposure and QCA is about data organisation. GIE and QCA may be combined in a multi method evaluation design, as are RCT and QCA.
Kim Forss concluded by saying that the seminar was adding to the complexity of aid evaluation that is already spawning 2-300 evaluations a year of Swedish aid in addition to the 10-12 studies produced by EBA. In fact less than 10 % of aid evaluations use the methods discussed during the session. My conclusion is that there is ample room for expanding the range of approaches and methods used and that EBA would do well to continue stimulating the discussion of evaluation procedures related to aid. We need to expand the range of methodological approaches to fully use the evaluation tool to increase our understanding and knowledge of the processes and results of international aid.
Eva Lithman