Evidence, Sources and Critical Appraisal

PCS-S04 · Module 3 — Evidence and Sources · Lesson 3.1 · Study time: 70 minutes · Language: English

Learning objectives

By the end of this lesson you will be able to:

Identify the main classes of evidence available for the study of security sector governance and what each can and cannot establish.
Appraise a source for reliability, credibility, provenance and relevance, treating those as separate judgements.
Detect the common defects in evidence used in this field, including circular sourcing and measurement drift.
Cite and record evidence so that another person could reconstruct your reasoning independently.

Key terms

Provenance: The traceable history of a piece of evidence: who produced it, when, for what purpose and through what chain of hands. Evidence without provenance is testimony without a witness.
Reliability and credibility: Reliability is a property of the source over time; credibility is a property of the specific report. A generally reliable source can file an incredible report, and an unreliable source can be right.
Circular sourcing: Several apparently independent reports that all trace back to one original. It is the single most common way a weak claim acquires the appearance of corroboration.
Measurement drift: A change in how something is counted that produces an apparent change in the thing counted. Always check whether the instrument changed before concluding that the world did.
Relevance: Whether the evidence bears on the question you are actually asking. Highly reliable evidence about the wrong question is worth nothing, and is dangerous because it feels solid.

Lesson

What counts as evidence here

This lesson. Evidence, Sources and Critical Appraisal belongs to Module 3 — Evidence and Sources of PCS-S04 Security Sector Governance, taught in the School of Peace, Conflict and Security Studies. Work through the written lesson first, then the slide deck below, then the activity. Every element on this page is live HTML text: it can be enlarged, printed, searched, read aloud by assistive technology and machine-translated without loss.

Work on security sector governance draws on several distinct classes of evidence: conflict event datasets, peace agreement texts, truth commission reports, field interviews with affected communities, humanitarian needs assessments and UN and regional body documentation. These are not interchangeable. Each is produced by an institution with its own purposes, and those purposes shape what gets recorded, in what categories and with what silences. Administrative records exist to administer, not to answer research questions; they are excellent for the questions the administration cared about and systematically blind elsewhere. Interviews recover meaning and motive but are constructed in the encounter and reshaped by memory.

The practical consequence is that the choice of evidence class is a substantive analytical decision, not a matter of convenience. The strongest work in peace, conflict and security studies usually triangulates: it takes a claim established in one evidence class and asks whether it survives contact with a different one that has different blind spots. Where two independent classes converge, confidence rises legitimately. Where they diverge, the divergence is itself the most interesting finding in the study and should be reported rather than smoothed away.

Appraising a source

Appraisal means answering four separate questions and refusing to collapse them. What is the provenance — who made this, when, and why? How reliable is the source in general, judged on its record rather than on its prestige? How credible is this particular report, judged on internal consistency, specificity and the presence of detail the source had no reason to invent? And is it relevant to the question actually being asked? A source can score well on three and fail on the fourth, and the fourth failure is fatal.

Keeping the four judgements separate matters because they get conflated in practice, almost always in the direction of prestige. An institution with a strong reputation is assumed to be reliable, its report is therefore assumed to be credible, and its relevance goes unexamined. The discipline of writing the four judgements down separately — even in a single line each — catches this. Where a grading vocabulary is available in your field, use it consistently rather than inventing adjectives.

How evidence goes wrong

Circular sourcing is the first and most damaging defect. Five reports say the same thing; they look like corroboration; four of them are citing the fifth, or all five are citing a sixth that no longer exists. The remedy is mechanical: for every claim that matters, follow the citation chain to its origin and record where it terminates. In security sector governance this exercise frequently ends at a single unpublished estimate, a press release or a figure whose original context was quite different.

Measurement drift is the second. An indicator is redefined, a reporting threshold is lowered, a data collection instrument is improved, and the resulting change in the series is read as a change in the world. The third defect is survivorship: the cases that reach the record are the ones that survived some prior filter, so the record describes survivors and is silent about the rest. The fourth is timing — evidence gathered after an outcome is known is contaminated by that knowledge, and retrospective accounts of why a decision was taken are among the least reliable material in this field.

Recording and citing

Your evidence record should allow another competent person to reconstruct your reasoning without speaking to you. That means, for each item: the full citation, the date accessed, the specific page or section, the claim it supports, your appraisal judgements, and any doubt you hold. Keep it as you go. Reconstructing an evidence trail after the analysis is written is slow, and it tends to be reconstructed in a way that flatters the conclusion.

Cite the source you actually used. If you read a claim in a review rather than in the original study, cite the review and say so. Second-hand citation is not a sin; concealed second-hand citation is, because it transfers to your work an authority that neither you nor the review has earned, and it propagates errors. Where a source is behind a paywall or is not publicly retrievable, say that too, so a reader knows what they cannot check.

Worked example: tracing a widely repeated figure

A striking statistic about security sector governance circulates widely. It appears in a news report, a policy brief and two academic papers. Begin by opening all four and reading their citations rather than their text. The news report cites the policy brief. The policy brief cites one of the papers. The first paper cites the second. The second paper cites a conference presentation from several years earlier. That presentation, when found, gives the figure as an illustrative projection based on assumptions stated on the slide.

At that point you know something valuable. The figure has one origin, it was never an observation, and the assumptions behind it have not been revisited since. This does not necessarily mean it is wrong. It does mean that the apparent weight of four independent sources is an illusion, and that anyone building a recommendation on that number is building on a single unverified estimate. Reporting this — carefully, with the chain documented — is a genuine contribution, and it takes a competent researcher about ninety minutes.

Slide deck

A ten-slide presentation of this lesson, built in HTML so that it prints, translates, enlarges and reads aloud on any device.

Slide 1 of 10