Introduction | Complexity | Theory-driven inquiry | Interesting papers | Bibliography | About | Download | Workshop

Theory-driven inquiry in the field of evaluation and research

Theory-driven evaluation

During the 1980s, theory-driven evaluation emerged in the discipline of evaluation. Chen & Rossi developed it as an answer to policy and programme evaluation approaches that remained limited to before-after and input-output designs or that focused narrowly on methodological issues (method-driven evaluation).

Theory-driven evaluation aims at assessing not only the effectiveness of an intervention but also its causal mechanisms, taking into account the context of the intervention.

The central element of this approach is the programme theory. For any intervention, a programme theory can be described. This theory explains how the planners expect the intervention to reach its objective. Describing this often implicit set of assumptions allows to understand what is being implemented and why (not). The programme theory represents a hypothesis that can be tested and further developed.

Theory should in this case be understood as middle-range theories or theories of the middle range as defined by R.K. Merton.

These are “theories that lie between the minor but necessary working hypotheses (…) and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organisation and social change”.

"Programmes are theories incarnate"
(N. Tilley)

Recommended reading

  • Chen, H.-T. & Rossi, P. (1987) The theory-driven approach to validity. Evaluation and Program Planning, 10, 95-103.
  • Chen, H.-T. (1990) Theory-driven evaluations, Newbury Park, California, Sage Publications.

The 2 parts of a programme theory
For Chen & Rossi, the programme theory consists of 2 parts: the normative theory and the causative theory. Later, Chen called this respectively the action model and the causal model.

The normative theory (the action model)

The normative theory presents the theoretical background that informs the design and implementation of the intervention and spells out the objectives and the implementation procedure.

It describes how the actual intervention was different or not from the planned intervention and the way the intervention was actually implemented.

It also describes the actual outcomes compared to the intended outcomes, and seeks not only to document positive but also negative outcomes.

Assessment of the normative theory can help decision makers, since it evaluates not only the effectiveness, but also the consistency of the implementation of the programme. This allows distinguishing programme theory-failure from implementation failure and provides feedback for improving the intervention.

The causative theory (the causal model)

The causal theory specifies the underlying causal mechanisms in terms of relationships between intervention and outcome, influence of context and intervening factors.

Unearthing the causal theory opens the box between the intervention and its outcomes. It allows judging on the value of the intervention and assessing whether the assumed causal processes actually did take place. It allows unpacking the intervention into its components and to check which element was most important and which intervening factors are most critical.

 

Theory of change

The Theory of Change (ToC) approach was developed by the Roundtable on Community Change (the Aspen Institute, USA). More pragmatic in its approach, ToC was developed to evaluate community-based programmes that typically involve many agencies and actors, have several levels and strands of activities, objectives and strategies that shift in time, and outcomes that are difficult to measure.

It is similar to theory-driven evaluation in that it seeks to establish the links between interventions, contexts and outcomes.

This is done by developing and testing ‘logic models’ that describe the populations that are targeted, the indicators used to monitor change, the thresholds of indicators that indicate significant change and the time lines. They specify short-, medium- and long-term outcomes to map how the intervention leads to the expected outcome and to enable attributing change to (parts of) the intervention.

These models are developed in collaboration with the stakeholders, ideally during the planning phase of a project during so-called ‘surfacing exercises’.

All models are wrong but some are useful
(Kernick 2005)

Recommended reading

  • Connell, J. P., Kubisch, A., Schorr, L. & Weiss, C. (1995) New approaches to evaluating community initiatives. Concepts, methods, and contexts, Washington D.C., The Aspen Institute.
  • Judge, K. & Bauld, L. (2001) Strong theory, flexible methods: evaluating complex community-based initiatives. Critical Public Health, 11, 19-38.
  • Mason, P. & Barnes, M. (2007) Constructing theories of change. Methods and sources. Evaluation, 13, 151-170.

 

Realist evaluation and synthesis

Realist evaluation can be considered as an approach within theory-driven evaluation although it has stronger philosophical underpinnings. It proposes conceptual tools to apply the principles of theory-based evaluation.

Realist inquiry intends to answer the question: “What is it about this programme that makes it work, for whom and in what circumstances?”, or in other words: which mechanisms cause which outcome under which conditions?

Realist inquiry has an explanatory focus and aims in essence at unravelling mechanisms of change. Based on critical realism, it considers that interventions work (or not) because actors take up what is offered by the intervention (or not). The interaction with the actors and with specific context elements triggers mechanisms, which cause certain outcomes to occur.

In this approach, the theory that provides the starting point is called the “Middle Range Theory”.

Board with post-it notes

 

Realist evaluation

The CMO configuration

In realist evaluation, the “Context-Mechanism-Outcome” configuration describes how outcome patterns emerge from the interaction between intervention, context and mechanisms.

The context is made up by the circumstances within which public health interventions are implemented. It includes the stakeholders, their interests and convictions regarding change and the process of implementation, and the organisational, socio-economic, cultural and political conditions.

Mechanisms are the drivers of the reactions of the target group, which lead to change and which are triggered by the intervention within a certain context.

Realist evaluation in practice

Evidence building using a realist perspective implies searching for CMO-configurations:

At the start, the middle range theory that underlies the intervention is made explicit: these are the assumptions held by the people involved regarding the expected outcome of an intervention and how - through which mechanisms or reactions - this will be achieved. This is complemented by existing (published) knowledge and experience.
Mixed methods are used to collect data about the elements of this initial middle range theory (intervention, outcomes, mechanisms, context).
During data analysis, context-mechanism-outcome configurations are identified and tested for plausibility.
The initial middle range theory is adapted in the light of the findings, which then becomes the starting point of new studies.

Recommended reading

  • Pawson, R. & Tilley, N. (1997) Realistic Evaluation, London, Sage.
  • Pawson, R. & Tilley, N. (2001) Realistic Evaluation Bloodlines. American Journal of Evaluation, 22, 317-324.
  • Kazi, M. (2003) Realist evaluation in practice. Health and social work, London, Sage Publication Ltd.

Realist synthesis

Pawson applied the principles of realist evaluation to the synthesis of evidence. Realist synthesis aims at informing policymakers through detailed ‘lessons learned’ by providing information on effectiveness but also on the mechanisms that cause the effect and the context conditions that are required to make the intervention work. This allows policymakers to better decide whether interventions can be expected to have the same results in their own setting.

This approach assesses findings of research and evaluations for evidence on the interactions between context, mechanisms and outcome. It aims at providing plausible explanations as to how interventions have produced their results, in which conditions and for which groups of the population.


Recommended reading

Realist evaluation and critical realism

Many authors refer to critical realism when discussing the philosophical roots of realist evaluation. Understanding the core elements of realism may help in applying the principles of theory-driven inquiry, and especially clarify how causality, mechanisms and context can be examined.

Critical realism

Critical realism is a particular philosophy of science developed by Ray Bhaskar, Rom Harré, Andrew Sayer and others. Adherents of realism maintain that social phenomena are real world objects and that these are not contingent on human observation: they do not only exist in people’s minds and are therefore independent of the researcher’s claims about them. In other words, social objects and the conjunctions between them exist and therefore can be described.

 

However, at the same time, knowledge is socially and historically constructed. Realists maintain that constructs that explain individual and social life are to be derived from observable events.

Realism bases this explanation on the perspective of ‘generative causality’. It explains change brought about by programmes by referring to the actors who act and change (or not) a situation under specific conditions and under influence of external events (including the intervention). The actors and the interventions are embedded in a stratified social reality, which is the result of an interplay between individuals and institutions, each with their own interests and objectives.

Recommended reading

  • Dickinson, H. (2006) The evaluation of health and social care partnerships: an analysis of approaches and synthesis for the future. Health Soc Care Community, 14, 375-83.
  • Connelly, J. (2007) Evaluating complex public health interventions: theory, methods and scope of realist enquiry. Journal of Evaluation in Clinical Practice, 13.

Key elements of critical realism

Of main interest to us are the views of critical realism on stratified nature of the real world, the generative nature of causality, the interaction between social structure and agency, and retroduction, critical realism’s approach of learning.

1. Critical realism stratifies reality in the ‘empirical’ (the day-to-day experience), the ‘actual’ (observable patterns) and the ‘real’ (the relatively unchanging existence, which can be disclosed by science). The ‘real’ encompasses the ‘actual’, which in turn includes the ‘empirical’. What can be observed is only part of what actually is, which in turn is only the actualised part of what is real.
2. Recognising the social world as a dynamic open system and the importance of social action, critical realism aims at identifying the generative mechanisms that underlie the social reality. Causality is generative: actors and society have potential mechanisms of causation by their very nature, which are actualised when an event or intervention combined with the right context factors trigger these generative mechanisms.

These generative mechanisms are not directly observable but they are real.
3. Critical realism considers that reality comes about as a result of an interaction between agency and structure (the transformational model of social activity). Generative mechanisms are actualised through the interplay between actors and structures. Such mechanisms can be positive and facilitate change or counteracting and block change.  
4. Retroduction is the technique through which generative mechanisms can be identified: through iterative abstraction, an argument moves from a description of some phenomenon to a description of something that produces it or is a condition for it.

These principles have informed realist evaluation. For Pawson and Tilley, the exploration of the underlying generative mechanisms should reflect the embeddedness of the intervention/programme in the social reality, the micro- and macro-level processes of change and how actors’ choices and use of their resources lead to the outcome. The CMO configuration offers a practical method to explore and analyse these generative mechanisms.

Recommended reading

  • Archer, M., Bhaskar, R., Collier, A., Lawson, T. & Norrie, A. (1998) Critical Realism. Essential Readings, London & New York, Routledge.
    Connelly, J. (2000) A realistic theory of health sector management. The case for critical realism. Journal of Management in Medicine, 14, 262-271.
  • McEvoy, P. & Richards, D. (2003) Critical realism: a way forward for evaluation research in nursing? J Adv Nurs, 43, 411-20.
  • Mingers, J. (2000) The contribution of critical realism as an underpinning philosophy for OR/MS and systems. Journal of the Operational Research Society, 51, 1256-1270.

Advantages of theory-driven inquiry

Adherents of theory-driven inquiry argue that it has several advantages:

It is well suited to investigate change in complex social systems: its focus on the generative causality that underlies interventions offers an approach to unravel the root causes and mechanisms of change that take into account the interaction between agency and structure and emergence of behaviours and change.

Evaluations that build on the programme theory allow the evaluator to identify the intermediate steps of the (hypothetical) causal chain. This can help in sharpening the focus of the evaluation.

If the programme theory is made explicit together with the main actors, it can lead to a better, shared understanding of the intervention. This in turn could improve the ownership and lead to more context-adapted interventions.

Theory-building helps to overcome the low external validity and low power to explain change of traditional case studies and process evaluations: it indicates in which conditions and how the effects were obtained. As such it improves transferability of findings to other settings.

Recommended reading

  • Birckmayer, J. D. & Weiss, C. H. (2000) Theory-based evaluation in practice. What do we learn? Eval Rev, 24, 407-31.
  • Calnan, M. & Ferlie, E. (2003) Analysing process in healthcare: the methodological and theoretical challenges. Policy and Politics, 31, 185-93.
  • Costner, H. (1989) The validity of conclusions in evaluation research: a further development of Chen and Rossi's theory-driven approach. Evaluation and Program Planning, 12, 345-353.
  • Pawson, R. & Tilley, N. (2006) Realist Evaluation. DPRN Thematic Meeting 2006 Report on Evaluation. Development Policy Review Network.

 

< prev
next >

    Download

     

    TDI workshop


    On 22 and 23 November 2010, the Unit of Health Care Management, Department of Public Health (ITM), organised an international expert meeting on theory-driven evaluation.

    About 15 researchers presented research and evaluations and discussed the methodological challenges of realist evaluation, theory-driven evaluation and realist synthesis in health systems research.

    In total 30 researchers, policymakers and programme managers attended the workshop.

    About this website

    This theory-driven inquiry website is developed by the Health Care Management Unit of the Department of Public Health, Institute of Tropical Medicine, Antwerp and the Development Policy & Practice Unit of the Royal Tropical Institute, Amsterdam. We collaborate on the development and implementation of theory-driven inquiry in health systems research, and aim at stimulating a better understanding of the advantages of this approach among both researchers and policymakers. To this end, we are engaged in a number of research and evaluation studies that use TD inquiry principles.

    Institute of Tropical Medicine

    Unit of Health Care Management

    TD inquiry project evaluations
    Passage
    Défi Jeunes

    TD inquiry research projects
    FemHealth
    PhD Bruno Marchal
    PhD Pierre Blaise

    The cluster Research methodology for complexity in healthcare
    Presentations

    Publications

    Contact
    Bruno Marchal - personal page
    Guy Kegels - personal page
    Tom Hoerée
    Josefien Van Olmen - personal page
    Sara Van Belle

     

     

     

    Royal Tropical Institute

    Department of Development, Policy and Practice

    TD inquiry project evaluations

    Realist evaluation of ASHA program in a district (India)

    TD inquiry research projects

    Realist inquiry in capacity building
    PhD Marjolein Dieleman

    Publications
    Research and evaluation methodology

    Contact
    Marjolein Dieleman
    Barend Gerretsen
    Sumit Kane
    Prisca Zwanikken