aes19 International Evaluation Conference

Mohammad Alatoom (Office of Environment and Heritage), Emily Prentice (Office of Environment and Heritage), Larissa Brisbane (Office of Environment and Heritage)

Research and knowledge generation is often in the 'too hard' basket for evaluation, being viewed as a public good, a foundational activity, difficult to value economically, or combination of these. Historically, academia has valued 'research' through metrics such as impact factors, publishing records, citations and successful funding applications. While these indicators reflect academic interest in the research, they do not reveal much about the fulfilment of other end users' needs.

If evaluation judges the 'merit, worth or value' of a thing, then evaluation of targeted research activities should fully consider how the outputs and outcomes advance and enrich our knowledge, enabling more informed decision-making. The evaluation should ideally demonstrate to what extent the research provides value to end users, as well as capture any distant outcomes for peripheral end users. How, then, do we evaluate research beyond traditional academic indicators? How do we evaluate the impact and effectiveness of research programs that are in progress or yet to report findings? And how do we best engage and involve end users in research evaluation, from planning to monitoring and final execution?

We present a best practice review and its application through a case study to examine these questions in a practical context. We describe evaluation planning for a targeted research program that is designed to generate insights into a complex problem, while satisfying the needs of a diverse range of end users. We discuss integrating evaluation planning into program design, engaging end users in developing the evaluation framework, the challenges of establishing KPIs for research evaluation, and reflect briefly on capturing the longer-term outcomes and options to apply economic valuation methods.