The potential for a Data Science and Natural Language Processing approach in evaluation

A case study of meta-evaluation in the field of International Development Cooperation.

The potential for alternative methods to collect and analyze data for evaluation, besides the more traditional methods such as interviews, surveys and focus groups are believed to increase with intensified digitalization and increased social engagement on the internet.

This study aims to probe the possibility for using data science applications in meta evaluations of development cooperation programs. The proposed approach will involve a range of analytical methods with an emphasis on computer-based processing of human language, or so-called Natural Language Processing (NLP).

The study will address the following questions:

  1. Can a data science and NLP approach produce reliable assessments of what past evaluations have concluded about aid projects and programmes relating to OECD/DAC’s evaluation criteria relevance?
  2. What are the strengths and weaknesses of these methods compared to approaches relying on manual techniques?

Authors: Gustav Engström, PhD, data scientist, Jonas Norén, M.Sc., consultant, Cecilia Ljungman, senior evaluation advisor

Reference group chair: Torgny Holmgren

Project manager at EBA: Lisa Hjelm

Expected publication: Q2 2021