aes19 International Evaluation Conference

Jasper Odgers (ARTD Consultants), Klas Johansson (ARTD Consultants)

We will tell you how Natural Language Processing (NLP) can be used to reduce time and costs associated with qualitative analysis by up to 75%. Our experience with this technology will allow for a vibrant discussion about the real benefits of machine-assisted qualitative analysis. The ethics, limitations and future directions of the technology will also be discussed.

This technology can be used to analyse large amounts of unstructured text data in a way that reduces the resource burden of analysing large qualitative datasets. By using techniques such as topic modelling and keyword identification, analysts can interpret the contents of large datasets in a fraction of the time it would take to do manually. Improvements in this technology will have profound impacts on the practice of evaluation as the use of the technology becomes more widespread. Much of the analysis work that was a large part of an evaluator’s job will be able to be done quickly and easily by machine-assisted technology; however, we focus on the continued need for humans to be involved throughout the analysis process. NLP is also adept at identifying themes from data which may not be apparent to human analysts. Integrating this technology with ongoing monitoring data means that evaluators don’t need to constantly analyse incoming data but can easily keep up to date and concentrate on interpretation and innovative reporting.

As the technology improves and becomes more widespread it is inevitable that it will have an impact on how evaluations are designed and therefore the theory which underpins them.