Learning objectives
By the end of this lesson you will be able to:
- Describe the main frameworks used to structure professional analysis of aceh studies.
- Select an appropriate framework for a given problem and justify the choice against alternatives.
- Apply Analysis of Competing Hypotheses to a real case and produce a defensible output.
- Recognise the failure modes of framework-driven work, including false rigour and premature closure.
Key terms
- Analysis of Competing Hypotheses
- Tests evidence against every plausible hypothesis rather than the favoured one, and looks for evidence that discriminates.
- Key Assumptions Check
- Surfaces the unexamined beliefs holding a judgement together and asks what would falsify them.
- The Admiralty (NATO) grading scale
- Separates the reliability of a source from the credibility of the specific report.
- ICD 203 analytic standards
- Require objectivity, independence, timeliness, sourcing, distinguished judgement and expressed uncertainty.
- Indicators and warning matrices
- Convert a scenario into observable, dated signposts that can be monitored.
Lesson
What a framework is for
This lesson. Session 14 — Academic and Practical Skills for Aceh Studies belongs to Module 2 — Frameworks and Standards of KBA Aceh Studies, taught in the School of Intelligence Studies. Work through the written lesson first, then the slide deck below, then the activity. Every element on this page is live HTML text: it can be enlarged, printed, searched, read aloud by assistive technology and machine-translated without loss.
A framework is a disciplined way of not fooling yourself. It does not generate conclusions; it forces you to make your reasoning visible so that someone else can find the weak joint. In intelligence and strategic analysis, the value of Analysis of Competing Hypotheses is not that it produces a better answer than an expert’s intuition, but that it produces an answer whose basis can be inspected, challenged and revised when new evidence arrives. Intuition, however good, leaves no audit trail, and in aceh studies an unauditable judgement is close to useless once it is contested.
This also explains why frameworks are unpopular with confident practitioners. They are slower, they surface disagreement that could have been left buried, and they often end by widening rather than narrowing the range of defensible conclusions. All three of those properties are features. A method that always confirms what the senior person in the room already thought is not a method; it is a ritual.
The main frameworks in this field
Five frameworks recur in professional work on aceh studies. Analysis of Competing Hypotheses tests evidence against every plausible hypothesis rather than the favoured one, and looks for evidence that discriminates. Key Assumptions Check surfaces the unexamined beliefs holding a judgement together and asks what would falsify them. The Admiralty (NATO) grading scale separates the reliability of a source from the credibility of the specific report. ICD 203 analytic standards require objectivity, independence, timeliness, sourcing, distinguished judgement and expressed uncertainty. Finally, Indicators and warning matrices convert a scenario into observable, dated signposts that can be monitored. Each of these has a natural domain and a boundary beyond which it produces confident nonsense.
Choosing between them is a real analytical decision and should be recorded as one. The relevant questions are: what kind of uncertainty dominates this problem, how much time do I have, who has to be persuaded, and what will happen if I am wrong? A framework that is excellent for a six-month research project may be unusable in a forty-minute decision meeting, and the professional skill is to run a compressed but honest version rather than to abandon structure altogether.
Applying a framework properly
Take Analysis of Competing Hypotheses. Applied properly it has four steps. First, state the question in a form that has more than one possible answer — questions with only one admissible answer are advocacy, not analysis. Second, lay out the alternatives explicitly, including the one you find least attractive, and give each a fair statement in its own strongest terms. Third, list the evidence and note against each item which alternatives it supports, which it weakens and, crucially, which it does not discriminate between at all. Fourth, write the conclusion together with the specific evidence that would overturn it.
The fourth step is the one most often skipped and the one that carries most of the value. An analysis that cannot say what would change its mind is not falsifiable and therefore cannot be tested by events. Recording the falsifier also converts your product into a monitoring instrument: if the named condition appears, the conclusion is due for revision, and someone can be assigned to watch for it.
How frameworks fail
The first failure is false rigour, in which the apparatus of a method is displayed while the judgements inside it remain unexamined. A neatly formatted matrix populated with guesses is not more reliable than the guesses; it is less reliable, because the format signals a confidence the content does not support. The second failure is premature closure, in which the framework is used to justify a conclusion that was reached before the work began. The tell is that the alternatives are stated weakly and the discriminating evidence all happens to point one way.
The third failure is scope error — applying a framework outside the conditions it assumes. The Admiralty (NATO) grading scale, for example, is built for situations in which the relevant categories are reasonably stable; used in a fast-moving situation it will impose an order that is not there. The remedy in every case is the same and is unglamorous: state your assumptions before you start, name the conditions under which the framework would not apply, and check at the end whether those conditions held.
Worked example: running Analysis of Competing Hypotheses under time pressure
A colleague asks for a view on aceh studies by the end of the day. There is no time for a full application, so run the compressed version. Spend ten minutes writing the question and three genuine alternatives. Spend twenty minutes listing every piece of evidence you already have and marking, for each, whether it discriminates between the alternatives or is consistent with all of them. In practice a large share of the available evidence turns out to be non-discriminating, and simply seeing that is often the most useful thing you can tell your colleague.
Spend the final ten minutes writing three sentences: the judgement, the confidence level with the reason for it, and the one development that would change the assessment. That output is short enough to be read, honest about its limits, and testable against what happens next. It is also markedly better than the alternative product of the same forty minutes, which is a fluent paragraph asserting the first plausible answer with unearned certainty.
Slide deck
A ten-slide presentation of this lesson. Every slide is plain HTML, so it prints, translates and reads on any device.
Click a number to jump to a slide, or scroll the deck sideways.
Activity
Apply Analysis of Competing Hypotheses to a live problem in Aceh Studies
Choose a current, genuinely unresolved question in aceh studies. Apply Analysis of Competing Hypotheses in full. Produce a one-page output containing the question, at least three alternatives each stated in its strongest form, an evidence table marking which items discriminate and which do not, your judgement with an explicit confidence level, and the single development that would most quickly overturn it.
Expected output. A one-page structured analysis with an evidence table and a named falsifier.
Suggested time. 120 minutes
Check your understanding
Q1. What is the main argument for using a framework rather than expert intuition?
Not that it is more accurate in every case, but that it leaves an audit trail. A structured judgement can be inspected, challenged and revised when new evidence arrives; an intuitive one cannot, which makes it very weak as soon as it is contested.
Q2. What is false rigour and how would you detect it in a submitted analysis?
It is the display of methodological apparatus over unexamined content — a neat matrix filled with guesses. Detect it by asking, for any single cell, what evidence supports that entry. If there is no answer, the format is carrying confidence the content has not earned.
Q3. Why should an analysis state what would change its mind?
Because a conclusion that cannot be falsified cannot be tested by events, and because naming the falsifier converts the analysis into a monitoring instrument that tells someone what to watch for.
References and further reading
- Heuer, R. J. (1999) Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, CIA.
- Heuer, R. J. and Pherson, R. H. (2020) Structured Analytic Techniques for Intelligence Analysis. 3rd edn. Thousand Oaks: CQ Press.
- Lowenthal, M. M. (2019) Intelligence: From Secrets to Policy. 8th edn. Thousand Oaks: CQ Press.
- Office of the Director of National Intelligence (2015) Intelligence Community Directive 203: Analytic Standards. Washington, DC: ODNI.
- Johnson, L. K. (ed.) (2010) The Oxford Handbook of National Security Intelligence. Oxford: Oxford University Press.
- Kahneman, D. (2011) Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.
- Gigerenzer, G. and Gaissmaier, W. (2011) ‘Heuristic decision making’, Annual Review of Psychology, 62, pp. 451–482.
Use of AI. You may use generative AI to summarise, translate or critique the material in this lesson, provided you verify every factual claim and citation against the sources listed above. Assessed work must carry a use-of-AI statement.