aes19 International Evaluation Conference

Kale Dyer (Family & Community Services)

Program logics provide a framework for a systematic, integrated approach to program planning, implementation, and evaluation. They foster a shared understanding of how a program operates by clearly articulating program activities and desired outcomes, and clearly illustrating the change processes underlying an intervention.

This presentation will demonstrate an extension of program logic focusing on better integrating evidence, making explicit the mechanism of change, and embedding the NSW Human Services Outcomes Framework into program design and evaluation. A distinguishing feature of the approach is the inclusion of sections that articulate the research evidence and mechanisms of change for the program. The approach includes the evidence base for how and why the core components and flexible activities that make up the program are expected to achieve the proposed outcomes. The ability to generalise program findings is improved by identifying core components and flexible activities. These evidence extensions highlight why components of the program are likely to be effective, and links client needs to intended outcomes. This clarification facilitates improved commissioning of research and evaluation, embedding evidence in programs, explicit discussion of mechanisms of change, and a client centred approach to achieving outcomes.

Discussion around the benefits and challenges of implementation of this extended program logic model in a government agency will be provided. Benefits include how it has facilitated more effective program evaluations by identifying areas of focus, informing the development of meaningful evaluation questions and identifying relevant client centred measures to address those questions.

Georgia Marett (ARTD Consultants), Jack Cassidy (ARTD Consultants)

This presentation shares insights into how behavioural economics and Behavioural Insights (BI) are used in program and service design and explores ways in which evaluation can and should take BI into account. A critical concept discussed in this paper is cognitive load. Research shows that cognitive overload can negatively impact decision-making and lead to more shallow processing of information and poor information retention. One method by which BI improves program decision-making and evaluation quality is by increasing the cognitive capacity of individuals.

We illustrate how service design can take cognitive load and BI into account and what might happen if BI are ignored when designing programs. Then, we examine and explain how to evaluate programs which have incorporated BI (including how cognitive load can be incorporated into a logic model, monitoring and evaluation frameworks and/or key evaluation questions). Finally, we conclude with a discussion about whether evaluation effectively uses the cognitive capacity of its stakeholders and practitioners.

This subject is important because while BI is a hot-topic in general and in evaluation, there is a lack of understanding about the ways in which it can be applied and how to evaluate those applications. Cognitive capacity is less well understood but is vital to understanding how to craft effective services, evaluate these services and conduct evaluations regardless of whether BI are included in the target of the evaluation. We will tie this into a realist perspective of evaluation through a discussion of how BI differ in effectiveness between people and situations.

Anne Markiewicz (Anne Markiewicz and Asssociates Pty Ltd), Susan Garner (Garner Willisson)

We are going to un-box an interactive game designed by two experienced presenters. The game will be Snakes and Ladders adapted to illustrate the ups and downs in the trajectory and life of the evaluator. Well designed assignments with realistic terms of reference and expectations and good stakeholder engagement will push the evaluator upwards in the game whereas ill-conceived, unrealistically scoped and politically challenged projects with hidden agendas and questionable stakeholder engagement will push the player downwards.

Topic for discussion: The fluctuating trajectory and experiences of the evaluator in conducting evaluation projects. This presentation has been influenced by the key note speaker from AES 2018 Karol Olejniczak 'Transforming evaluation practice with serious games'.

Purpose: To use game theory to illustrate how some evaluation projects go well and the success factors involved while others projects do not.
Participants should enjoy the interactive session and provide a forum for them to reflect on their experiences with evaluation projects. The session will highlight success factors and factors that get in the way of successful outcomes in evaluation projects.

Andrew Moore (NZ  Defence), Victoria Carling (NZ Defence, NZ)

Empathy mapping is an emerging collaborative approach that focuses on the results of a programme. Used to gain the perspective of different stakeholders, from the commissioner to the programme participants, it seeks to define what they truly value from a programme. Empathy mapping requires participants to reflect on what success looks like, according to them, by considering what they would see, hear, do, say, think, or feel during and post programme. The results can then be used, as the building blocks of evaluation rubrics to define measurable criteria. The collaborative approach ensures a shared understanding is achieved on the quality, value, and effectiveness of a programme.

Drawing from their experience the presenters will demonstrate how empathy mapping has been used to build the foundations for successful evaluation within NZ Defence. Highlighting how empathy mapping can maximize contact time with key stakeholders, document the shared understanding of programme results and subsequently promote a collective interpretation of evaluation reports.

The session will allow participants to gain an insight into: What is empathy mapping? Where did it come from? What are the components of an empathy map? Why are they useful as building blocks for evaluation practice? How they can be used to build evaluation-rubrics?

Anthea Rutter (The  University of Melbourne)

What does it mean to call ourselves an evaluator? How do we define our craft? Let me pose another question: what do you put on your departure card when leaving Australia? - Evaluator?

I once put evaluator on my departure card before a flight to the States. I then spent the best part of an hour in Los Angeles airport trying to explain to a customs official what exactly an evaluator is. I felt it would have been so much easier to be a Plumber, an Electrician or a Nurse. We can easily conjure up the visual - somehow, it's not the same for an evaluator. However, defining our craft is important so that others, whether they are emerging evaluators or clients will understand what we are about, as well as what we are not about.

The AES Fellows are an important resource for understanding the history of evaluation, how it has evolved as well as looking towards the future. During the last eight months or so, I have interviewed the majority of the AES Fellows to get their take on what it means to be an evaluator today. I was rewarded by an honest and reflective look at their careers and gleaned some ideas for those emerging evaluators. For a number of those early pioneers of evaluation, they came into it when it was a fledgling field when it was still in the throes of trying to define itself. It has emerged as a profession and has been strengthened by becoming multi-disciplinary as it recognises that it needs to draw on many fields.

In this short paper, I will present some of those thoughts and experiences of the AES Fellows, to illuminate the path, if possible, for new evaluators. I would hope to pass on some ideas which can assist in skill building as well as identifying the qualities needed for the evaluator of today. This paper should add to the knowledge base in terms of providing some valuable information on the perceptions of those evaluators who have gone before.

Jo Hall (Australian  National University)

There are a number of different methods for synthesising information across multiple evaluations. The emphasis of one of these, realist synthesis (Pawson and Tilley), is on identifying theory (context-mechanism-outcome configurations) to answer the question ‘what works for whom in what circumstances, in what respects and how?’ There are relatively few examples of realist synthesis and they sometimes struggle to articulate mechanisms and theory in ways that can be helpful for policy makers. In particular they tend to be insufficiently focused on explanation and to develop separate lists of context, mechanisms and outcomes. More examples of realist synthesis are important to grow the practical experience of using and refining the method. It is also important to demonstrate a viable and potentially more useful alternative to systematic reviews that are based on randomised control trials, for which there is a growing appetite.

I will share with you my PhD work which adopted a realist synthesis methodology for Australia’s Department of Foreign Affairs Review of Program Evaluations, to see what could be learned from the evaluation reports across two topic areas: policy influence and promoting gender equality.
I will briefly present the findings but spend most of this session reflecting on the methodology. The primary sources of information for the review were the 37 evaluation reports completed by program areas in 2017 and 14 interviews with program evaluators and DFAT staff. The method focused on coding explanatory text in evaluation reports and interview transcripts and analysing the coded text with the help of Nvivo software, drawing on substantive theory.

In a 20-minute presentation I will highlight key aspects of the process and my reflections on mid-range theory, mechanisms and explanation. The remaining 5 minutes will be for questions and discussion.

The learning papers are available at:
https://dfat.gov.au/aid/how-we-measure-performance/ode/strategic-evaluations/Documents/review-of-2017-program-evaluations-policy-influence-learning-paper.pdf

https://dfat.gov.au/aid/how-we-measure-performance/ode/strategic-evaluations/Documents/review-of-2017-program-evaluations-gender-learning-paper.pdf

Jenny Riley (Clear Horizon), Jess Dart (Clear Horizon), Kristy Mansfield (Seer Data and Analytics), Reuben Stanton (Paper Giant), Chris Newman (ArcBlue Asia Pacific)

Digital, Cloud, Data Science, AI and Machine Learning, Robots....what does all this mean for the field of evaluation? Award winning evaluator Jess Dart, will host a panel of experts to explore current and emerging trends in what is hailed the 4th revolution. We will explore how new technologies are being used for social change (phone apps for finding free food, wearables for tracking in aged-care facilitates, social media for building resilience amongst farmers, apps for stream lining fines applications) and what evaluators need to be equipped to evaluate these technological interventions and also how digital can be leveraged to enhance the practice of evaluation.

The panel will reflect on real-world examples of how technical fixes can fail but also how new technology and design approaches can more democratic, participatory, transparent and importantly useful at potentially much lower costs than before. The panel will share what they have seen work well and how they evaluate success. We will also explore the ethics, risks and challenges of digital data collection, storage and reporting. We will discuss big data, small data as well as open and closed data and how we can leverage digital.

Ghislain Arbour  (University of Melbourne)

Evaluation frameworks are currently an important concern in evaluation practice, especially for organisations who desire to organise their evaluation activities. But the reflections and decision in that domain are plagued with imprecisions and ambiguities regarding the constitutive dimensions of frameworks. This renders more difficult the identification of needs and potential answers in their selection or development.

In response, this paper provides a model to analyse frameworks for program evaluation organised around four dimensions. The model states that a framework for evaluation is an intellectual framework, made of concepts and or theories (first dimension: types of ideas) about an object related to evaluation (second dimension: object), where the said concepts and theories can be positive and/or normative (third dimension: analytical perspective). These three dimensions provide the means to describe, explain or judge an evaluation related matter. A fourth and optional dimension, the institutional character of a framework, allows an evaluation framework to become a form of regulation for behaviours related to program evaluation (fourth dimension: institutional dimension).

In essence, this paper will raise our awareness about the kinds of theoretical "boxes" we encounter in evaluation so we can get better at relying on them, and even turn them into influential policies when it counts.

Zarin Salter (Active Transport and Safety, Urban Mobility, Department of Transport - WA), Dr Kim Carter (Data Analysis  stralia, Pty Ltd)

Implemented by The Government of Western Australia, Your Move delivers a suite of tailored travel behaviour change (TBC) programs that provide participants with localised, personalised information, coaching conversations and ongoing feedback to encourage them to walk, ride a bike and use public transport more often for their daily trips.

In 2018, de-identified, residentially coded SmartRider ticketing data made it possible to analyse the public transport patronage habits of residents who lived in two previous Your Move project areas and statistically compare their travel with those who lived in areas of greater Perth that received no Your Move projects. The data source was representative of the whole metropolitan area and was still sufficiently large enough for analysis even after a thorough data cleaning process was applied.
The resulting figures for the two previous Your Move projects were impressive and the most reliable estimate of public transport mode shift that Your Move has been able to obtain in its 20-year history. Having robust figures for public transport mode shift made it possible to extrapolate the shift in other modes and model the overall benefits of a Your Move project to the whole community.

Traditionally, TBC programs have been evaluated using self-report data collection techniques which are expensive and prone to risks associated with data reliability, survey length and respondent burden, small sample size, inaccurate sampling between interviewers, control group selection, panel recruitment loss, and weather variability.

This presentation will discuss the need for practitioners to innovate in the TBC evaluation space, specifically with respect to data source accuracy, and will share insights learned from un-packing the box of treasures hidden within ticketing data.

Jasper Odgers (ARTD Consultants), Klas Johansson (ARTD Consultants)

We will tell you how Natural Language Processing (NLP) can be used to reduce time and costs associated with qualitative analysis by up to 75%. Our experience with this technology will allow for a vibrant discussion about the real benefits of machine-assisted qualitative analysis. The ethics, limitations and future directions of the technology will also be discussed.

This technology can be used to analyse large amounts of unstructured text data in a way that reduces the resource burden of analysing large qualitative datasets. By using techniques such as topic modelling and keyword identification, analysts can interpret the contents of large datasets in a fraction of the time it would take to do manually. Improvements in this technology will have profound impacts on the practice of evaluation as the use of the technology becomes more widespread. Much of the analysis work that was a large part of an evaluator’s job will be able to be done quickly and easily by machine-assisted technology; however, we focus on the continued need for humans to be involved throughout the analysis process. NLP is also adept at identifying themes from data which may not be apparent to human analysts. Integrating this technology with ongoing monitoring data means that evaluators don’t need to constantly analyse incoming data but can easily keep up to date and concentrate on interpretation and innovative reporting.

As the technology improves and becomes more widespread it is inevitable that it will have an impact on how evaluations are designed and therefore the theory which underpins them.

Carolyn Hooper (Allen and Clarke Policy and Regulatory Specialists)

We have all kinds of tools at our fingertips, yet many of us under-utilise them. If you have NVivo and want to get more out of it, learning how to develop a template in MSWord is a good way forward. In five minutes, I will show you how to to it, and you will never look back.

Kahiwa Sebire  
It might look like just fun and games, but Lego Serious Play (LSP) is a powerful facilitation tool to enable groups to surface deeper-level assumptions about a topic or program. By supporting participants to think metaphorically to build and then communicate their idea or viewpoint, groups can achieve stronger and clearer communication.

Let me share an example of how I used LSP to help a team build a shared vision of success, while uncovering competing assumptions in a safe and structured manner, and ideas for how you could use it to construct program theories, define success criteria, gather participant insights.

Takara Tsuzaki  (Western Michigan University)

This presentation demonstrates a case study of a mixed method and bilingual program evaluation which was conducted on a newly launched rural medicine/rural generalist program in Japan with a focus on collaborative and iterative learning processes. The client, GENEPRO LLC and the evaluator will share challenges in designing and implementing the evaluation, and how we have been successful in building trust among stakeholders, integrating evaluation into practice, and fostering iterative learning within the organization.

The model - Rural Generalist Program Japan (RGPJ) - is based on the Australian model which has been regarded as the most comprehensive and matured rural generalist medicine training scheme in the world. To meet the specific needs of rural generalist medicine in Japan, provision of rural healthcare was needed to be tailored to regional and local context. Exporting this medical training scheme from Australia to Japan also meant a new collaborative endeavor to develop a unique program evaluation model and approach in Japan.

This presentation will highlight the contextual differences between the East and the West in terms of philosophies and cultural values and how they are manifest in the evaluation practices. The concept of both the theoretical and practical evaluation has developed differently in Japan in the past 50 years when it is compared to the West. Furthermore, evaluation has been conducted predominantly using quantitative data in the medical and healthcare sector in Japan. However, the rural generalist medicine requires distinctly broad scope of practice as well as unique combination of abilities and aptitude to respond to the community needs of rural and remote areas of Japan. As a result, the evaluation approach, including the underlying values, philosophies and methodologies had to be thoroughly examined and openly discussed to bring all the stakeholders on board.

We will share the lessons from the collaborative evaluation process by discussing: what the evaluative thinking and collaborative evaluation design mean in the Japanese rural and medical settings; how we have come up with innovative approaches to communicate with stakeholders who have evaluation anxiety and fear of modernist undertaking; how we have acknowledged and overcome (in)translatability issues in languages, imbedded values, and social contexts of each stakeholder groups; and how the collaborative evaluation processes impacted the organizational culture during and after the evaluation.

Gill Westhorp  (Charles Darwin University)

What do we know? What can we know, and how do we know that we know it? These are philosophical questions with real implications for the practice of evaluation. Epistemology is the branch of philosophy dealing with the nature of knowledge. Different epistemologies underpin different approaches in research and evaluation. They have implications for what data is considered to be 'valid', how data can or should be collected, how data is analysed and interpreted, and under what conditions findings are portable to other contexts.

This paper deals with two epistemologies - realist and constructivist - from a realist viewpoint. Some authors have claimed that realists 'are realists ontologically, but constructivists epistemologically'. That is, realists believe that there is a real world, which exists independently of our interpretations of it ("realist ontology"). However, we all construct our own interpretations of it. Knowledge is not a direct representation of reality, but an interpretation of it, constructed in our own heads, and shaped by language, culture, personal experience, and previous learning ("constructivist epistemology"). Knowledge does not exist independently of 'the person who knows'. In radical constructivism, we cannot even be sure that there is a real world. Perhaps we are all just avatars in some giant computer game.

This paper argues that there are areas of overlap, but also areas of distinction between, realist and constructivist epistemology. These distinctions have implications for evaluation practice. It will briefly describe the key assumptions of constructivism, and contrast these with key assumptions in realism. It will use a hypothetical evaluation as an example to discuss differences in: the purposes of constructivist and realist investigation; the nature of the data that is collected; the ways that analysis is undertaken; how 'valuing' is approached and how evaluation adds value; the nature of findings; and the portability of findings.

Amy Bestman (Health Equity Research & Development Unit (HERDU), Sydney Local Health District), Jane Lloyd (Health Equity Research & Development Unit (HERDU),  Sydney Local Health District), David Lilley (Health Equity Research & Development Unit (HERDU), Sydney Local Health District), Barbara Hawkshaw (Central and Eastern Primary Health Network)

This presentation wrestles with the balance between ensuring a robust community-led, inter-sectoral, public health program in a culturally and linguistically diverse (CALD) location and how to effectively provide sufficient monitoring, evaluation, reflection and improvement opportunities while the intervention is in situ.

Can Get Health in Canterbury (CGHiC) is a unique inter-sectoral program with three key partners (the University of New South Wales, Sydney Local Health District and Central Eastern Primary Health Network) and many local partnerships with community organisations. It was established in 2013 to address high health needs among CALD population groups within Canterbury, NSW.
CGHiC's partnership with the community is supported by the employment of community networkers and the establishment of collective control projects. Bengali and Arabic networkers link the community with the health system, and also provide insight to the health system on the unique needs of the community. The collective control projects enable the community to have greater power over decision making, priority setting and allocation of resources. These projects aim to improve capacity of both community groups and the health system and encourage bi-directional learning and reflection.

Two external evaluations have previously been conducted which provide a point in time reflection on the impact of the project. Now that CGHiC is in its sixth year of operation, we are evaluating the program in-house with the following foci: the external impact of the program; the governance structure, priority setting and decision making of the program; and, the activities of the program. While this process is ongoing, the program team have implemented monitoring tools and processes to measure recent activities. The CGHiC evaluation will contribute to the field of evaluation through the development of novel methodologies, approaches and insights to evaluating complex place-based, multi-sectoral, population-level programs in situ.

Mathea Roorda (Allen  + Clarke)

Values are fundamental to evaluation as they provide the basis against which evaluative judgments are made. Yet evaluators often overlook them. In this skill building session, participants will be introduced to a framework intended to unbox dimensions of value for publicly-funded programs. As the overall conference theme states: evaluation can be gift - it has the potential to strengthen people's lives. Evaluation also comes with responsibilities, one of which is that the evaluator's judgments need to be based on all relevant values, not just those of the evaluation commissioner. The framework draws on two approaches to valuing, one of which comes from a branch of philosophy that is focused on value (how we understand concepts such as good and bad); the second on describing value as understood by different program stakeholders. We will step through the framework's components and then discuss its applicability for evaluation practice. A handbook for using the framework will be made available to participants.

Liz Norsa (Western Sydney University), Nathan Jones (Aboriginal Health Unit SWSLHD), Karen Beetson (Aboriginal Health Unit SWSLHD), An  Speizer (Aboriginal Health Unit SWSLHD), Raylene Blackburn (Camden & Campbelltown Hospitals SWSLHD), Ilse Blign lt (School of Medicine Western Sydney University)

Aboriginal people with chronic conditions are more likely to leave hospital with incomplete transfer of care arrangements and more likely to be readmitted after a recent hospitalisation. The Aboriginal Transfer of Care (ATOC) Program at South Western Sydney Local Health District (SWSLHD), in which Aboriginal Liaison Officers and Transfer of Care nurses work as team to deliver a holistic patient-centred model of care, was designed to address this problem by ensuring consideration of an Aboriginal patient’s medical, cultural and psychosocial needs. Promising early results led to a formal evaluation funded by NSW Health under its Translational Research Grant Scheme. SWSLHD, Western Sydney University and the Ministry of Health are partners in this mixed-methods evaluation. The qualitative evaluation component aimed to: document the program model, describe what is ‘successful’ transfer of care for patients, their families and service providers, and identify opportunities for program enhancement and extension. The evaluation employed participatory methods, which involved over 40 interviews, participant observation and workshops at two hospitals. SWSLHD and the university members of the evaluation team brought insider and outsider perspectives: Aboriginal and non-Aboriginal; service manager or provider, and evaluator. This short presentation describes how the evaluation approach and ways of working were shaped by these different perspectives.

Andrew Hawkins (ARTD  Consultants)

This paper is about identifying, managing and mitigating the risk that a program will not produce its intended effects. A principle of this approach is that a program at its core is simply a proposition that a certain course of action will lead to a certain set of outcomes. It is about putting the logic back in program logic.

Program Design Logic (PDL) is a tool for developing evidence based policy and programs. Through the language of 'necessary' and 'sufficient' conditions in place of 'outputs' and 'outcomes' it provides a framework to determine if a program or course of action makes sense 'on paper' before we attempt to determine if it makes sense in 'reality' through monitoring and evaluation.

The five types of risk are

1. It doesn't make sense on paper - logical risk
2. It makes sense on paper, but assumptions don't hold - assumption risk
3. It makes sense on paper, but we didn't do what we said we would do - performance risk
4. It makes sense on paper, assumptions hold, we do what we said we would do, but outputs don't materialise - theoretical risk
5. It makes sense on paper, assumptions hold, we do what we said we would do, outputs materialise, but intended outcomes don't follow, so the array of outputs was not actually sufficient to bring about a desired future state - logical risk
6. It makes sense on paper, assumptions hold, we do what we said we would do, outputs materialise, intended outcomes follow, but longer term outcomes don't materialise -external factor risk

This paper will discuss how a PDL approach can provide a comprehensive risk management framework before the first participant is even enrolled, which may then be managed and mitigated through program re-design as well as adaptive monitoring and evaluation.

Rini Mowson (Clear Horizon), Sarah Leslie (Clear Horizon)

The context within which an evaluand exists matters in evaluation. The AES evaluators' professional learning competency framework dedicates an entire domain to "attention to culture, stakeholders and contexts". Oakley, Pratt and Clayton (1998) argued that evaluation context should be treated as being at the very heart of social development and impact assessment must take full account of the bigger picture in arriving at the conclusion about the success or failure of social development programs. Thus, adapting and managing the evaluation "context" is important to balance ensuring sustainable and impactful evaluation to the end users/beneficiaries, with satisfying the needs of the program team and/or evaluation commissioner.

This paper will seek to answer two questions:: "What are the domains of context that evaluators need to be aware of?" and "How can evaluators adapt their practice to fit the context where they work?'.
The presenters will draw on their experiences in evaluation in multicultural contexts through their work in international development.

The presenters propose three domains of context that evaluators should consider before embarking on an evaluation journey. Firstly, studies demonstrate the importance of applying basic principles of evaluation such as participation, community empowerment and communicating the evaluation results back to beneficiaries, however most evaluations are donor driven exercises. With this limitation, how can evaluators empower funding recipients to enforce the application of basic principles of evaluation. Secondly, how can evaluators address power dynamics in the evaluation process to ensure the evaluation results will represent the real outcomes of the program achieved across different types of beneficiaries. Thirdly, presenters propose that all evaluation should find ways to ensure evaluation will support capacity building of relevant stakeholders including beneficiaries and communities.

Florent Gomez (NSW Department of Finance, Services and Innovation)

In the same way that some countries use culture as a soft power approach to extend their influence, evaluation should give serious consideration to soft capacity building tools such as Communities of Practice. This approach can be incredibly effective in diffusing evaluative thinking across organisations that are less familiar with it.

A New South Wales government department which is not a traditional stronghold for evaluation – as compared to human services departments such as education, health or community services – has established a successful Evaluation Community of Practice since November 2017. The Community of Practice brings together staff with varying levels of evaluation maturity to ‘share the love for evaluation’. The intent is to offer a more informal and less intimidating forum for participants to share challenges and learning than a traditional expert-to-learner approach. Over 50 people gather at each quarterly event where presenters provide case studies, panel discussions and practical exercises such as collectively developing a program logic or crafting good survey questions.

After a year and a half, participants reported an increased understanding of what evaluation is about and of key tools such as program logic, as well as applying those learning back in their workplace. The Community of Practice has opened up the conversation on evaluation across the organisation. While a slow and diffuse process, there is now a growing interest in evidence-based approaches, outcome framing and evaluative thinking.

Other soft power approaches used involve staff mentoring and evaluation advice. These have proved to be particularly powerful in improving the quality of evaluations – and are not necessarily much more resource intensive than formal training. Provided at the initial stage, targeted evaluation advice contributes to getting the evaluation framing right which generates a better evaluation brief. This, in turn, results in better evaluation outcomes, where the evaluation produces evidence around what the organisation is interested to learn about.

Janice Smith (Charles Sturt University), Shaarn Hayward (Charles Sturt University), Suellen Priest (Charles Sturt University), Christine Lindsay (Charles Sturt University)

The Indigenous Academic Success Program at Charles Sturt University offers a suite of academic services to Indigenous students to improve aspiration, retention, and success. The program has supported over 730 students enrolled across Charles Sturt University courses since its conception in 2016 and is largely comprised of Indigenous staff, who oversee the planning, evaluation, implementation, reporting, and improvement of the program.

The program is deeply embedded within the Indigenous community, with six of the seven permanent staff currently employed in the program identifying as Indigenous, and representing ten different Indigenous nations or language groups. Evaluative practices have been applied throughout program setup and delivery, through the use of program logics, quantitative and qualitative methodologies. A key evaluation method is the use of interviews to gather feedback from students who are using, have used, or been offered access to the service.

This presentation unpacks the program's evaluation process and design, detailing the ways the programs annual evaluation report determines the progress, outcomes, and development of the program in consideration of the student community the program works with. It will also consider how the evaluation of both participants and those invited to participate who did not take up the offer of support has been conducted in a way that provides a safe and effective mechanism for Indigenous participants to participate. Feedback from students at all levels of engagement is positioned as central to understanding the program's progress and success, and this presentation will look at how the evaluation data has been used to measure the program's progress in reaching outcomes and inform its improvements and future direction.

Keren Winterford (University of Technology Sydney), Anna Gero (Institute for Sustainable Futures, University of Technology Sydney), Jake Phelan (Austalia Volunteers Program)

This paper explores the practice of a formative evaluation for the Australian Volunteers Program and sets out why formative evaluation is valuable to setting program focus and defining approaches to impact evaluation. Reflections from independent evaluators and the Monitoring Evaluation and Learning team of the Australian Volunteers Program are provided within this presentation drawing together multi-stakeholder and practitioner perspectives on theory and practice of formative evaluation.

The overall objective of the formative evaluation presented in this paper was to map the global footprint of the Australian Volunteers Program in three impact areas in order to (i) establish a baseline; (ii) inform strategic options for strengthening engagement in the impact areas and; (iii) propose methodology for demonstrating outcomes in impact areas. The three impact areas of Inclusive economic growth; Human Rights; and Climate Change/Disaster Resilience/Food Security are informed by the Australian Government Volunteers Program Global Program Strategy. Rather than setting out evaluation findings, the paper explores the practice of collaborative evaluation design; use of mixed methods including key informant interviews, document review, and quantitative analysis to prepare working definitions of impact areas. We explore the practice of drawing on local (country contexts) and global measures (Sustainable Development Goals) to define impact areas and how we have made sense of these to apply to the Australian Volunteers Program.

The paper distinguishes the theory and practice of formative evaluation and sets out the unique contribution it offers to policy and programming agendas. The paper talks about the value of evaluation across multiple points in the project cycle and value of linking formative and summative evaluations as highlighted within this case. Informed by this case study, the presenters offer tips and tricks for those commissioning and conducting evaluations to ensure formative evaluations provide best contribution to policy and programming agendas.

Matt Healey (First Person Consulting)

Typically, evaluators are seen and presented as all-knowing experts across a never-ending range of areas: quantitative, qualitative and mixed research methods, engagement approaches, cultural competencies, reporting tools and platforms. On top of this, is a need to understand the constant change within and across social, health and environmental arenas, exponential changes in technology and the implications for evaluation. In many ways, it is impossible for evaluators to know everything - even more so for evaluators at the earlier stages of their career.

During an emerging evaluators panel session at aes18, one of the key themes that emerged when emerging and early career evaluators discussed how to move out of the intermediate "fuzzy middle" towards becoming 'experts' was the need to be both comfortable in uncertainty and confident with themselves, their knowledge and their practice in evaluation. While the AES competency framework emphasises competence in a range of areas, the need to be confident (and develop confidence) is only implicit across domains, and only explicit in the context of building confidence in others, and in statistical methods!

This session will draw on practices and principles from the presenter's own experience in developing his confidence in the context of presenting, facilitating and dealing with large audiences. Through a mix of lightning talks, light-hearted hands-on activities and reflective small group discussions attendees will leave with tools and approaches immediately implementable during and post-conference. Ultimately, this highly interactive skill session will make explicit the unspoken (but crucial) soft skill of confidence.

Mark Galvin (EY)

Recent government moves towards outcomes budgeting is the latest illustration that outcomes thinking is here to stay. Outcomes evaluation coupled with economic evaluation is increasing and increasingly interdependent, especially in the social policy and services space. With such anticipation, the risk of an empty box looms large. Demonstrating and valuing outcomes requires intentional and fit-for-purpose measurement approaches. Sharing approaches is critical to further innovation and support for robust public decision making.

This Ignite presentation will showcase changes in the policy landscape, as well as visual depictions of evaluation methodologies that situate 'traditional' social outcomes as benefits and how significant economic value is derived through effective services delivery and cost avoidance.

Carolyn Hooper (Allen and Clarke Policy and Regulatory Specialists)

During an intervention evaluation, front-line service delivery staff made periodic diary entries using an on-line portal. Diarists responded to prompts specific to the evaluation questions. The output provided valuable insights to the day-to-day realities of those delivering the intervention; resulting in front-line staff having a strong voice in the evaluation report. The e-diary is an accessible, innovative method for collecting data, suited to situations where a detailed view of the work at the intervention delivery interface is valuable, but direct observation by an evaluator is problematic. Come and see how we did it.

Adrian Field (Dovetail), Julian King (Julian King and Associates), Kate McKegg (The Knowledge Institute)

How do project innovations create changes in wider organisational systems and practice? This short paper will discuss our learning from evaluating three dynamic road safety projects working within an innovation umbrella programme.

This session will highlight the challenges and opportunities for taking innovation to scale, reflecting on our learning from theoretical approaches outside evaluation that offer compelling new windows for evaluator’s understanding of impact and change. Grounded in real-world application of three innovative road safety projects, the paper will present the inter-weaving of socio-technical systems theory, developmental evaluation, rubrics, and learning from the innovation literature.

These approaches were used, along with their practical application through rubrics and multiple data collection methods, to explore the extent to which the projects fostered innovation that translated into sustained business operations.

This paper will provide useful ideas and reflections for participants including how collaboratively developed evaluation rubrics were used to define and assess levels and dimensions of system change that each project could reflect against, through a developmental process of engagement and reflection.

We will also reflect on the contribution that collaborative partnerships, communities of practice, people-centred approaches, and reframing risk offer to evaluation practitioners as avenues for exploring the translation of innovation to system change. Socio-technical systems theory will be provided as a lens for understanding the potential for local or niche innovations to lever changes in wider systems.

The session will conclude with an exploration of the role of evaluation in capturing and catalysing innovation.

Liam Downing (Centre for Education Statistics and Evaluation)

In a session designed especially for those who LOVE watching those unboxing videos on YouTube, I will unbox, set up, and use a brand new evaluation capacity building role live on the AES 2019 stage. I will show you what's inside, how it works and what it can do. You can see if it's the right choice for you to build skills and grow the profession through capacity building. This Ignite presentation will also use props. PROPS!

Racheal Norris (GP Synergy), Linda Klein (GP Synergy)

This presentation will outline the processes and challenges involved in developing an efficient evaluation framework, using a state-wide vocational training organisation as a case-study. GP Synergy delivers an accredited General Practice training program, across eight highly diverse subregions of NSW and the ACT, for doctors wishing to specialise as General Practitioners. A small Evaluation Team was established in 2017 to develop a rigorous, adaptive evaluation system to monitor and report on delivery of educational activities.

Using evidence-based methodology, the team adopted a participatory approach and engaged stakeholders across three key levels:

Education Executive
An interactive program logic workshop was held to discuss and identify various evaluation priorities at the senior-level.

Medical Educators
The team worked closely with individual educators to design evaluation tools that were standardised, yet responsive to the unique needs of each region. This involved careful consideration of psychometric properties to ensure robust and reliable measures of key outcomes. A semi-automated reporting system was created to maximise efficiency of delivering timely feedback, and the team guided educators to correctly interpret and utilise this information for continuous improvement.

GP Registrars
The team consulted with registrars (trainees) to explore and develop pathways to "close the loop" and communicate evaluation findings and implications for the training program. This also involved educating registrars about the broader theoretical framework behind evaluation and how to provide useful, constructive feedback.

Evaluation at GP Synergy remains an evolving process, with ongoing multi-level engagement ensuring evaluation systems continue to be responsive and adaptable to stakeholder needs. The role of the Evaluation Team in educating stakeholders and colleagues about evaluation 'steps' has been fundamental to successful data collection and reflection on findings resulting in change. Insights will be offered to others developing evaluation frameworks/methods within settings where flexibility and responsiveness are key.

Alison Laycock (University Centre for Rural Health), Gillian Harvey (The University of Adelaide), Nikki Percival (University of Technology Sydney), Frances Cunningham (Menzies School of Health Research), Jodie Bailie (University Centre for Rural Health), Veronica Matthews (University Centre for Rural Health), Kerry Copley (Aboriginal Medical Services Alliance Northern Territory), Louise Patel (Aboriginal Medical Services Alliance Northern Territory), Ross Bailie (University Centre for Rural Health)

Implementation science examines what methods and strategies work to promote the use of research findings and other evidence into routine practice, to improve the quality and effectiveness of health services and care. It explores, for example, how health interventions can be adapted and scaled in ways that are accessible and equitable to improve health. Implementation science can provide important knowledge for improving Aboriginal and Torres Strait Islander health, however little research addresses how implementation theories or frameworks have been applied to evaluate projects and programs in Indigenous health.

Drawing on developmental evaluation data, we used the integrated Promoting Action on Research Implementation in Health Services (iPARIHS) framework to examine factors contributing to the success, or otherwise, of a large-scale interactive dissemination project. The project engaged stakeholders with continuous quality improvement data from Aboriginal and Torres Strait Islander primary health care services to co-produce knowledge for improving care.

In this presentation, we describe how we selected and applied this theoretical framework as an evaluation tool. We examine the extent to which use of the framework enhanced our understanding of project interactions, limitations and success in the Aboriginal and Torres Strait Islander health care context and influenced our ongoing work to improve health.

This presentation is a merge of the two listed below and contains a free-ranging discussion on evaluator competencies,


Advanced Tips For Commissioning and Managing High-Quality, Useful Evaluation 
Jane Davidson (Real Evaluation LLC),Tessie Catsambas (Encompass LLC)

What are the most important traps to avoid and tips for commissioning and managing high-quality, value-for-money evaluation? This interactive panel session will be an informative helicopter tour for evaluation commissioners, evaluation team leaders, and internal and external professionals who oversee or manage evaluation projects. It will provide: (1) a deeper appreciation of the role of evaluation management in commissioning and delivering high-quality, value-for-money evaluations; (2) an overview of the role and essential competencies of evaluation managers; and (3) sample strategies and tools for commissioning and managing better and more useful evaluations for organizational learning and stronger leadership. Participants are invited to share their own experiences and engage in a highly interactive discussion with the presenters, who will draw on decades of practical experience leading both large international multi-country evaluations and small-team and solo evaluation projects, as well as providing advice to client organizations on how to scope, commission, and manage highly effective evaluations.

Unpacking the competencies - in theory and practice
 
Amy Gullickson (University of Melbourne Centre for Program Evaluation), Delyth Lloyd (Department of Health and Human Service), Sue Leahy (ARTD)

The AES Professional Learning Competency Framework was developed in 2012 and in 2019, the Learning and Professional Practice Committee engaged the AES community in research with the intention to update the competency set. The goal was to assess the framework in relation to what evaluation theorists have discussed in the literature about what skills and knowledge are needed for evaluation practice. In this interactive session, we'll report out on recent theoretical work in this area, the findings to date of the community research project, get community feedback on the findings so far and their relevance to evaluation practice, and discuss next steps.

Carmen Huckel Schneider (University of Sydney), Sarah Norris (University of Sydney), Sally Wortley (University of Sydney), Angus Ritchie (University of Sydney), Fiona Blyth (University of Sydney), Adam Elshaug (University of Sydney), Andrew Wilson (University of Sydney)

There have been increasing calls to take a systems-thinking approach to evaluating health policies and programs - acknowledging the complexity of health systems and the many actors, institutions, relationships, drivers and values that impact on health system change. Several key frameworks have emerged that support systems-thinking, including "WHOs Framework for Action"; "NASSS - Non-Adoption, Abandonment, and Challenges to Scale-Up, Spread and Sustainability"; and the "Vortex Model". However little has been written on how to operationalise systems framework elements into practical evaluation studies comprising methodologically rigorous data collection and analysis methods - all while staying true to the principles of systems-thinking.

In this presentation we seek to unbox the challenge of operationalising a system-thinking approach to evaluating healthcare delivery innovations. We use the NASSS framework as our example to demonstrate how to expand system-thinking frameworks, progress towards theories and pose systems-thinking-driven, yet researchable questions. This requires crossing epistemological boundaries, and taking a 'multiple studies' approach adopting various methods of inquiry. We report on applying these principles to evaluate HealthPathways Sydney, a website for GPs to navigate care pathways for their patients through primary and specialist care. We followed a two phase approach, beginning with a series of sub-studies using standard qualitative and quantitative methods and reflected on the conduct of these studies to pinpoint system level factors (macro contexts, institutional settings, critical events, agents and relationships) that were necessary to understand in order to determine how the innovation interacted with the system. Our second phase adopted systems-thinking study methods including geo-spatial mapping, social network analysis, process tracing, frames analysis and situational analysis. Results were then synthesised into a rich case of the introduction of an innovation into the system. We uncovered progress towards desired outcomes, but also barriers to consolidating and embedding the technology when other system factors were in play.

Kathryn Dinh (UNSW), Heather Worth (UNSW), Bridget Haire (UNSW)

The field of evaluation tends to be dominated by certain Western-derived understandings of the way the world works and underlying belief that these understandings are universal. Culturally responsive evaluation recognises the existence of diverse world views and some of its exponents argue that it needs to encompass more than simply working closely in collaboration with locally-based partners. It should additionally involve modifying and creating new evaluation approaches that are grounded in non-Western world views.

While there has been significant innovation in evaluation approaches that reflect Indigenous world views in Australia, New Zealand, the US and elsewhere, there has been less progress in reflecting the world views of South-East and East Asia. Buddhism has a significant global influence today, and particularly in these regions where it is practised by a large majority of the population.

In this presentation, we suggest an applied approach to culturally responsive evaluation by first analysing the world views underpinning Buddhism and the Most Significant Change (MSC) technique, a participatory method for monitoring and evaluation that involves the collection of stories of significant change. We then identify where these converge and diverge and suggest practical ways in which the MSC technique could be adapted to reflect a Buddhist world view. Finally we look at how in a globalised world, societies are made up of a complex and dynamic mix of values, philosophies, traditions, religions and cultures. We discuss that as evaluators, we can use this approach to work with locally-based colleagues to unpack the theory and value systems underpinning existing evaluation methods, and repackage the methods or create new ones that reflect, and are responsive to, the complex and dynamic world views in the local context being evaluated.