aes19 International Evaluation Conference

Patricia O'Connor (Australian Healthcare Associates), Tracey Marriner (Australian Healthcare Associates), Shantanu Sheshgir (Australian Healthcare Associates), Jill Waddell (Australian Healthcare Associates)

Assessing the incremental achievements of place-based-initiatives (PBIs) has become an increasingly important component of contemporary evaluation practice. While much is known about the characteristics of successful PBIs, the practicalities of assessing implementation progress across multiple PBIs in a single program remain a complex challenge.

When tasked with evaluating a national program jointly funded by the Australian Government Departments of Health and Education and Training, aimed at improving Aboriginal health and education outcomes, this challenge became a reality for our evaluation team.

This presentation explores the four-stage process undertaken to develop a tool to assess implementation progress across a 13-site PBI program. These sites included a mix of urban, regional and remote locations. PBI maturity ranged from several months to multiple years, with some sites adopting a collective impact approach.

In Stage 1, a literature scan was undertaken to identify the attributes of successful PBIs and the breadth of indicators/measures used to evaluate them. Stage 2 involved mapping each indicator/measure against the eight PBI domains identified in Stage 1. Duplicates were removed and multiple codes were applied in some cases to facilitate measurement by sub-themes such as collective impact and integration. Indicators/measures were then converted to a plain language statement format, so that achievements could be assessed using a five-point scale, ranging from 'not yet started' (0) to 'achieved' (4). A rubric was constructed from the literature findings to guide the rating process undertaken by the evaluation team (Stage 3).

Summing up ratings by PBI domain (Stage 4) identified the domains where a site had made achievements and domains that required a concentrated effort going forward. This standardised tool also facilitated reporting of program-level findings and insights.

Factor analysis will later be used to determine the most important indicators within each domain, thereby reducing the number of questions being asked.