Lesson 2.1 — Building a Theory of Change

PG-P06 · Module 1 — Core Concepts · Lesson 1.2 · Study time: 70 minutes · Language: English

Learning objectives

By the end of this lesson you will be able to:

Define the core concepts that organise monitoring, evaluation, and policy learning and distinguish them from their everyday near-synonyms.
Explain why the way a concept is defined determines what counts as evidence for or against it.
Identify the main points of disagreement between competing accounts of monitoring, evaluation, and policy learning.
Apply the vocabulary of public policy and governance accurately in written analysis.

Key terms

Agenda setting: A foundational idea in public policy and governance. Treat it as a working definition to be tested against cases rather than as a settled fact.
Policy instruments: Frequently confused with the term above in everyday usage. The distinction between them does real analytical work and is examined in this lesson.
State capacity: A concept that becomes visible mainly when it fails, which is why so much of the literature is built on failure cases.
Implementation gap: A contested term. Different schools attach different criteria to it, and a great deal of apparent disagreement in this field is disagreement about this word.
Operational definition: The version of a concept that specifies what you would actually observe or measure. Without one, a claim about monitoring, evaluation, and policy learning cannot be tested.

Lesson

Why definitions are not preliminaries

This lesson. Lesson 2.1 — Building a Theory of Change belongs to Module 1 — Core Concepts of PG-P06 Monitoring, Evaluation, and Policy Learning, taught in the School of Public Policy and Governance. Work through the written lesson first, then the slide deck below, then the activity. Every element on this page is live HTML text: it can be enlarged, printed, searched, read aloud by assistive technology and machine-translated without loss.

It is tempting to treat the definitional section of a course as throat-clearing before the real material begins. In public policy and governance that instinct is wrong, and expensively so. How you define monitoring, evaluation, and policy learning determines what you will count as an instance of it, which determines what evidence you will gather, which determines what conclusion you can reach. A study that defines the phenomenon broadly will find it everywhere and conclude that it is a general condition; a study that defines it narrowly will find it rarely and conclude that it is exceptional. Both may be internally valid and they will contradict each other in public.

This is why the literature in this area contains long-running arguments that never resolve. Participants are not disagreeing about the world; they are disagreeing about the boundary of a category and reporting the results as if they were about the world. Your first analytical obligation, whenever you read a claim about monitoring, evaluation, and policy learning, is to find the operational definition being used. If it is not stated, that absence is itself a finding, and a serious one.

The core vocabulary

Four concepts do most of the work in this field: agenda setting, policy instruments, state capacity and implementation gap. Each has an everyday sense that is close enough to the technical sense to cause trouble. The everyday sense of agenda setting is loose and evaluative; the technical sense is specific and, in principle, checkable. Using the everyday sense while borrowing the authority of the technical one is the most common weakness in student writing at this level, and it is easy for a marker to detect.

The remaining vocabulary of the field — accountability, collective action problems, evidence use and policy learning — is layered on top of that base. Notice that these terms are not independent. They form a structure in which changing your account of one forces changes elsewhere. If you adopt a demanding definition of agenda setting, you will find that your account of collective action problems has to become more permissive to keep the picture coherent. Watching for that kind of forced trade-off is a reliable way to understand a theoretical position quickly.

Where the disagreements actually are

Strip away the terminology and most disputes in monitoring, evaluation, and policy learning reduce to three questions. First, is the phenomenon best understood as a property of individuals, of organisations or of systems? Second, is it something that can be measured directly, or only through proxies whose relationship to the underlying quantity is itself a theory? Third, is the appropriate response corrective, structural or political? Positions in this field can be located fairly accurately by their answers to those three questions, and doing so is more useful than sorting authors into named schools.

The mistake to avoid is assuming that one answer must be correct across all cases. In public policy and governance the level at which a phenomenon is best analysed often varies with the question being asked. A hospital, a ministry and a household can all be the right unit of analysis for a study of the same problem, and the researcher’s obligation is to justify the choice, not to pretend it was inevitable.

Using the concepts well

A concept earns its place in your writing if removing it would change your conclusion. Apply that test ruthlessly. Sentences that name three theorists and reach the conclusion you would have reached anyway are decoration, and experienced markers read them as padding. Sentences that use state capacity to explain why the obvious interpretation of the evidence is wrong are analysis, and they carry marks.

Precision also means being willing to say what a concept excludes. If your definition of monitoring, evaluation, and policy learning would classify almost any observed situation as an instance, it is not doing analytical work. Test your definitions by looking for a plausible near-case that falls outside them. If you cannot construct one, the definition is too broad, and you should tighten it before you go any further.

Worked example: pulling a definition apart

Suppose a report states that monitoring, evaluation, and policy learning has ‘improved significantly’ over five years. Before accepting or rejecting that, ask four questions in order. What operational definition is in use? What data source generates the numbers, and did that source change during the period? What is the comparison — improvement against the previous year, against a target, or against a comparable population? And what would deterioration have looked like in this measurement system, given that many indicators are constructed so that they can only rise?

Working through those questions usually produces one of three verdicts. The claim survives, in which case you now understand it far better than the person who accepted it uncritically. The claim survives in weakened form, which is the most common outcome and is a perfectly respectable finding. Or the claim turns out to be an artefact of a definitional or measurement change, which is the finding that most often gets a piece of student work noticed. None of this requires new data collection; it requires reading the methods section, which most readers skip.

Slide deck

A ten-slide presentation of this lesson. Every slide is plain HTML, so it prints, translates and reads on any device.

Slide 1 of 10