EBA has published a new report titled “Data Science Methods in Development Evaluation: Exploring the Potential.”
The report examines the potential of using data science and Natural Language Processing (NLP) in development evaluation. It looks at how such methods can be used produce reliable assessments of what past evaluations have concluded about aid projects and programmes (relating to OECD/DAC’s evaluation criteria relevance). It also discusses the strengths and weaknesses of these methods compared to approaches relying on manual techniques.
Main findings:
- Descriptive statistics can be collected rapidly and effectively using data science methods.
- The levels of accuracy of the statistics generated was in general in line with that of a manual assessment.
- Challenges occurred in more complex interpretations of results, such as whether projects were deemed to be sustainable or not. However, complex interpretations also varied in the manual, human interpretations.
- The authors conclude that the usability of using data science methods depends on available resources, requirements for transparency and replicability, and the need for a scaled-up analysis.
Authors:
Gustav Engström, PhD, data scientist
Jonas Norén, M.Sc., consultant