This report examines the potential of using data science and Natural Language Processing (NLP) in development evaluations. It looks at how such methods can be used to produce reliable assessments of what past evaluations have concluded about aid projects and programmes (relating to OECD/DAC’s evaluation criteria relevance). It also discusses the strengths and weaknesses of these methods when compared to approaches that rely on manual techniques.
Main findings
- Descriptive statistics can be collected rapidly and effectively using data science methods.
- The levels of accuracy of the statistics generated was generally in line with that of a manual assessment.
- Challenges occurred during more complex interpretations of results, such as whether projects were deemed to be sustainable or not. However, complex interpretations also varied in the manual, human interpretations.
- The authors conclude that the usability of using data science methods depends on available resources, requirements for transparency and replicability, and the need for a scaled-up analysis.