aes19 International Evaluation Conference

Bridgette Hargreave (Evaluation Unit), Angelina Bruno (Insights and Evaluation, Department of Industry, Innovation and Science)

Impact evaluations of government programs are becoming tougher, particularly when the program involves a broad spectrum of effects (social, economic and environmental), and the data is held by many different agencies. Australian Government departments are now experimenting with linking data sets to undertake innovative data analysis, research and evaluation.

This year, the Evaluation Unit in the Department of Industry, Innovation and Science trialled a new mixed method methodology for impact evaluations. To evaluate the impact of government programs at a regional level, we combined the analysis of cross-portfolio datasets with other regional data and qualitative research.

We employed out-of-the-box thinking to explore how we could use data from sources such as the Business Longitudinal Analysis Data Environment, the Household, Income and Labour Dynamics in Australia Survey, and the Australian Census. We then integrated insights from interviews, program reports and relevant academic research.

Presenters will explain why we trialled this methodology on an evaluation of Tasmanian Innovation and Investment Funds, and how it was applied. We will also demonstrate how a more complete picture of program outcomes and broader effects can be obtained through combining innovative data sources and techniques with a different way of thinking. We will discuss our approach to the complex issue of developing a counterfactual for the evaluation. And, as all of this was experimental and none of it was easy, we will also highlight the challenges faced by those involved, the solutions attempted, and the many lessons learned along the way.

Attending this presentation will give you insights into how major cross-portfolio datasets can be used to enrich impact analysis, and ideas on how such approaches could be applied to your own evaluations.