aes19 International Evaluation Conference

Carmen Huckel Schneider (University of Sydney), Sarah Norris (University of Sydney), Sally Wortley (University of Sydney), Angus Ritchie (University of Sydney), Fiona Blyth (University of Sydney), Adam Elshaug (University of Sydney), Andrew Wilson (University of Sydney)

There have been increasing calls to take a systems-thinking approach to evaluating health policies and programs - acknowledging the complexity of health systems and the many actors, institutions, relationships, drivers and values that impact on health system change. Several key frameworks have emerged that support systems-thinking, including "WHOs Framework for Action"; "NASSS - Non-Adoption, Abandonment, and Challenges to Scale-Up, Spread and Sustainability"; and the "Vortex Model". However little has been written on how to operationalise systems framework elements into practical evaluation studies comprising methodologically rigorous data collection and analysis methods - all while staying true to the principles of systems-thinking.

In this presentation we seek to unbox the challenge of operationalising a system-thinking approach to evaluating healthcare delivery innovations. We use the NASSS framework as our example to demonstrate how to expand system-thinking frameworks, progress towards theories and pose systems-thinking-driven, yet researchable questions. This requires crossing epistemological boundaries, and taking a 'multiple studies' approach adopting various methods of inquiry. We report on applying these principles to evaluate HealthPathways Sydney, a website for GPs to navigate care pathways for their patients through primary and specialist care. We followed a two phase approach, beginning with a series of sub-studies using standard qualitative and quantitative methods and reflected on the conduct of these studies to pinpoint system level factors (macro contexts, institutional settings, critical events, agents and relationships) that were necessary to understand in order to determine how the innovation interacted with the system. Our second phase adopted systems-thinking study methods including geo-spatial mapping, social network analysis, process tracing, frames analysis and situational analysis. Results were then synthesised into a rich case of the introduction of an innovation into the system. We uncovered progress towards desired outcomes, but also barriers to consolidating and embedding the technology when other system factors were in play.