The ‘CES Guide to’ series will attempt to lift the lid on some of the approaches we use for the work we do with agencies, service providers and government departments.
Over the past few years, CES has worked on a number of evaluations. Evaluation involves asking questions about the impact of a project, programme or service. It can help generate insights on what has changed for people and families, or help to identify areas where a service can be improved. The approach you take depends on the questions you are asking and the information you need.
This first ‘CES Guide to’ is about realist evaluations.
The realist evaluation approach acknowledges the fact that people are different, and the circumstances they live in are different too. It looks at the many different factors at play in determining the success of an intervention or policy, and asks “What works, in what circumstances, and for whom?”
A Theory of Change or programme theory is used in a realist approach to focus the evaluation on the key questions of whether the intervention works, in what circumstances and for whom. It is a way of defining long term goals and ways to achieve them. A theory of change is developed in partnership with the intervention stakeholders – for example, people involved in delivering or using a service – and is also informed by what is known from the research literature in order to figure out the aims of the intervention. You then work backwards to see how you can achieve these aims in smaller steps.
The realist evaluation approach is underpinned by three key concepts:
The real world of designing and delivering services for people and families is complicated. A realist evaluation helps you untangle a situation when there are potentially a lot of different explanations for why changes in outcomes might be occurring. Unlike a closely controlled research study (for example, a Randomised Control Trial), where the conditions and settings are known or tightly managed throughout, realist approaches are designed to get at the messiness of how an intervention works or doesn’t work in the real-world, such as health, education and other social care settings.
One of the benefits of a realist evaluation is that it is flexible. You can change, adjust and refine the evaluation as you gather and analyse your data. The important final stage in realist evaluations is the refinement of your programme theory.
One of the challenges you can face at the outset is having too broad a goal for your evaluation. A benefit of a realist approach is that it helps you understand how different factors contribute to the success of a change or intervention. It tells you about contribution rather than causality. You can use both quantitative data and qualitative data in a realist evaluation. In fact, the most useful realist evaluations use different forms of measurement at the same time, such as surveys and focus groups, to show the different factors that contribute to an outcome. However, a challenge that can arise from this is that communicating the impact may not be straightforward.
It is important to manage stakeholder expectations and to communicate your aims and approaches clearly with your team. Participation of all relevant stakeholders at the beginning of the evaluation process helps to ensure its success. An important part of the realist evaluation approach is breaking down barriers to participation and looking for solutions to potential issues, including creating positive and effective engagement spaces, taking account of power and control dynamics, and advocacy.
The CES Guide to Realist Evaluations is based on a CES presentation at the International Realist Evaluation Conference in February 2021.
Watch: Community Healthcare Networks in Ireland - the role of stakeholder engagement from a realist lens.
