aes19 International Evaluation Conference

Racheal Norris (GP Synergy), Linda Klein (GP Synergy)

This presentation will outline the processes and challenges involved in developing an efficient evaluation framework, using a state-wide vocational training organisation as a case-study. GP Synergy delivers an accredited General Practice training program, across eight highly diverse subregions of NSW and the ACT, for doctors wishing to specialise as General Practitioners. A small Evaluation Team was established in 2017 to develop a rigorous, adaptive evaluation system to monitor and report on delivery of educational activities.

Using evidence-based methodology, the team adopted a participatory approach and engaged stakeholders across three key levels:

Education Executive
An interactive program logic workshop was held to discuss and identify various evaluation priorities at the senior-level.

Medical Educators
The team worked closely with individual educators to design evaluation tools that were standardised, yet responsive to the unique needs of each region. This involved careful consideration of psychometric properties to ensure robust and reliable measures of key outcomes. A semi-automated reporting system was created to maximise efficiency of delivering timely feedback, and the team guided educators to correctly interpret and utilise this information for continuous improvement.

GP Registrars
The team consulted with registrars (trainees) to explore and develop pathways to "close the loop" and communicate evaluation findings and implications for the training program. This also involved educating registrars about the broader theoretical framework behind evaluation and how to provide useful, constructive feedback.

Evaluation at GP Synergy remains an evolving process, with ongoing multi-level engagement ensuring evaluation systems continue to be responsive and adaptable to stakeholder needs. The role of the Evaluation Team in educating stakeholders and colleagues about evaluation 'steps' has been fundamental to successful data collection and reflection on findings resulting in change. Insights will be offered to others developing evaluation frameworks/methods within settings where flexibility and responsiveness are key.