During the 1980s, theory-driven evaluation emerged in the discipline of evaluation. Chen & Rossi developed it as an answer to policy and programme evaluation approaches that remained limited to before-after and input-output designs or that focused narrowly on methodological issues (method-driven evaluation).

Theory-driven evaluation aims at assessing not only the effectiveness of an intervention but also its causal mechanisms, taking into account the context of the intervention.

The central element of this approach is the programme theory. For any intervention, a programme theory can be described. This theory explains how the planners expect the intervention to reach its objective. Describing this often implicit set of assumptions allows to understand what is being implemented and why (not). The programme theory represents a hypothesis that can be tested and further developed.

Theory should in this case be understood as middle-range theories or theories of the middle range as defined by R.K. Merton.

These are “theories that lie between the minor but necessary working hypotheses (…) and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organisation and social change”.

The normative theory (the action model)

It also describes the actual outcomes compared to the intended outcomes, and seeks not only to document positive but also negative outcomes.

Assessment of the normative theory can help decision makers, since it evaluates not only the effectiveness, but also the consistency of the implementation of the programme. This allows distinguishing programme theory-failure from implementation failure and provides feedback for improving the intervention.

The causative theory (the causal model)

The causal theory specifies the underlying causal mechanisms in terms of relationships between intervention and outcome, influence of context and intervening factors.

Unearthing the causal theory opens the box between the intervention and its outcomes. It allows judging on the value of the intervention and assessing whether the assumed causal processes actually did take place. It allows unpacking the intervention into its components and to check which element was most important and which intervening factors are most critical.

The Theory of Change (ToC) approach was developed by the Roundtable on Community Change (the Aspen Institute, USA). More pragmatic in its approach, ToC was developed to evaluate community-based programmes that typically involve many agencies and actors, have several levels and strands of activities, objectives and strategies that shift in time, and outcomes that are difficult to measure.

It is similar to theory-driven evaluation in that it seeks to establish the links between interventions, contexts and outcomes.

This is done by developing and testing ‘logic models’ that describe the populations that are targeted, the indicators used to monitor change, the thresholds of indicators that indicate significant change and the time lines. They specify short-, medium- and long-term outcomes to map how the intervention leads to the expected outcome and to enable attributing change to (parts of) the intervention.

These models are developed in collaboration with the stakeholders, ideally during the planning phase of a project during so-called ‘surfacing exercises’.

All models are wrong but some are useful
(Kernick 2005)

Realist evaluation can be considered as an approach within theory-driven evaluation although it has stronger philosophical underpinnings. It proposes conceptual tools to apply the principles of theory-based evaluation.

Realist inquiry intends to answer the question: “What is it about this programme that makes it work, for whom and in what circumstances?”, or in other words: which mechanisms cause which outcome under which conditions?

Realist inquiry has an explanatory focus and aims in essence at unravelling mechanisms of change. Based on critical realism, it considers that interventions work (or not) because actors take up what is offered by the intervention (or not). The interaction with the actors and with specific context elements triggers mechanisms, which cause certain outcomes to occur.

In this approach, the theory that provides the starting point is called the “Middle Range Theory”.

The CMO configuration

In realist evaluation, the “Context-Mechanism-Outcome” configuration describes how outcome patterns emerge from the interaction between intervention, context and mechanisms.

The context is made up by the circumstances within which public health interventions are implemented. It includes the stakeholders, their interests and convictions regarding change and the process of implementation, and the organisational, socio-economic, cultural and political conditions.

Mechanisms are the drivers of the reactions of the target group, which lead to change and which are triggered by the intervention within a certain context.

Realist evaluation in practice

Evidence building using a realist perspective implies searching for CMO-configurations:

At the start, the middle range theory that underlies the intervention is made explicit: these are the assumptions held by the people involved regarding the expected outcome of an intervention and how - through which mechanisms or reactions - this will be achieved. This is complemented by existing (published) knowledge and experience.
Mixed methods are used to collect data about the elements of this initial middle range theory (intervention, outcomes, mechanisms, context).
During data analysis, context-mechanism-outcome configurations are identified and tested for plausibility.
The initial middle range theory is adapted in the light of the findings, which then becomes the starting point of new studies.

Pawson applied the principles of realist evaluation to the synthesis of evidence. Realist synthesis aims at informing policymakers through detailed ‘lessons learned’ by providing information on effectiveness but also on the mechanisms that cause the effect and the context conditions that are required to make the intervention work. This allows policymakers to better decide whether interventions can be expected to have the same results in their own setting.

This approach assesses findings of research and evaluations for evidence on the interactions between context, mechanisms and outcome. It aims at providing plausible explanations as to how interventions have produced their results, in which conditions and for which groups of the population.

Many authors refer to critical realism when discussing the philosophical roots of realist evaluation. Understanding the core elements of realism may help in applying the principles of theory-driven inquiry, and especially clarify how causality, mechanisms and context can be examined.

Critical realism

Critical realism is a particular philosophy of science developed by Ray Bhaskar, Rom Harré, Andrew Sayer and others. Adherents of realism maintain that social phenomena are real world objects and that these are not contingent on human observation: they do not only exist in people’s minds and are therefore independent of the researcher’s claims about them. In other words, social objects and the conjunctions between them exist and therefore can be described.

However, at the same time, knowledge is socially and historically constructed. Realists maintain that constructs that explain individual and social life are to be derived from observable events.

Realism bases this explanation on the perspective of ‘generative causality’. It explains change brought about by programmes by referring to the actors who act and change (or not) a situation under specific conditions and under influence of external events (including the intervention). The actors and the interventions are embedded in a stratified social reality, which is the result of an interplay between individuals and institutions, each with their own interests and objectives.

Key elements of critical realism

Of main interest to us are the views of critical realism on stratified nature of the real world, the generative nature of causality, the interaction between social structure and agency, and retroduction, critical realism’s approach of learning.

1. Critical realism stratifies reality in the ‘empirical’ (the day-to-day experience), the ‘actual’ (observable patterns) and the ‘real’ (the relatively unchanging existence, which can be disclosed by science). The ‘real’ encompasses the ‘actual’, which in turn includes the ‘empirical’. What can be observed is only part of what actually is, which in turn is only the actualised part of what is real.
2. Recognising the social world as a dynamic open system and the importance of social action, critical realism aims at identifying the generative mechanisms that underlie the social reality. Causality is generative: actors and society have potential mechanisms of causation by their very nature, which are actualised when an event or intervention combined with the right context factors trigger these generative mechanisms.

These generative mechanisms are not directly observable but they are real.
3. Critical realism considers that reality comes about as a result of an interaction between agency and structure (the transformational model of social activity). Generative mechanisms are actualised through the interplay between actors and structures. Such mechanisms can be positive and facilitate change or counteracting and block change.  
4. Retroduction is the technique through which generative mechanisms can be identified: through iterative abstraction, an argument moves from a description of some phenomenon to a description of something that produces it or is a condition for it.

These principles have informed realist evaluation. For Pawson and Tilley, the exploration of the underlying generative mechanisms should reflect the embeddedness of the intervention/programme in the social reality, the micro- and macro-level processes of change and how actors’ choices and use of their resources lead to the outcome. The CMO configuration offers a practical method to explore and analyse these generative mechanisms.

Adherents of theory-driven inquiry argue that it has several advantages:

It is well suited to investigate change in complex social systems: its focus on the generative causality that underlies interventions offers an approach to unravel the root causes and mechanisms of change that take into account the interaction between agency and structure and emergence of behaviours and change.

Evaluations that build on the programme theory allow the evaluator to identify the intermediate steps of the (hypothetical) causal chain. This can help in sharpening the focus of the evaluation.

If the programme theory is made explicit together with the main actors, it can lead to a better, shared understanding of the intervention. This in turn could improve the ownership and lead to more context-adapted interventions.

Theory-building helps to overcome the low external validity and low power to explain change of traditional case studies and process evaluations: it indicates in which conditions and how the effects were obtained. As such it improves transferability of findings to other settings.