Jo Hall (Australian National University)There are a number of different methods for synthesising information across multiple evaluations. The emphasis of one of these, realist synthesis (Pawson and Tilley), is on identifying theory (context-mechanism-outcome configurations) to answer the question ‘what works for whom in what circumstances, in what respects and how?’ There are relatively few examples of realist synthesis and they sometimes struggle to articulate mechanisms and theory in ways that can be helpful for policy makers. In particular they tend to be insufficiently focused on explanation and to develop separate lists of context, mechanisms and outcomes. More examples of realist synthesis are important to grow the practical experience of using and refining the method. It is also important to demonstrate a viable and potentially more useful alternative to systematic reviews that are based on randomised control trials, for which there is a growing appetite.
I will share with you my PhD work which adopted a realist synthesis methodology for Australia’s Department of Foreign Affairs
Review of Program Evaluations, to see what could be learned from the evaluation reports across two topic areas: policy influence and promoting gender equality.
I will briefly present the findings but spend most of this session reflecting on the methodology. The primary sources of information for the review were the 37 evaluation reports completed by program areas in 2017 and 14 interviews with program evaluators and DFAT staff. The method focused on coding explanatory text in evaluation reports and interview transcripts and analysing the coded text with the help of Nvivo software, drawing on substantive theory.
In a 20-minute presentation I will highlight key aspects of the process and my reflections on mid-range theory, mechanisms and explanation. The remaining 5 minutes will be for questions and discussion.
The learning papers are available at:
https://dfat.gov.au/aid/how-we-measure-performance/ode/strategic-evaluations/Documents/review-of-2017-program-evaluations-policy-influence-learning-paper.pdfhttps://dfat.gov.au/aid/how-we-measure-performance/ode/strategic-evaluations/Documents/review-of-2017-program-evaluations-gender-learning-paper.pdf
