Analytical Frameworks and Standards

AI-A03 · Module 2 — Frameworks and Standards · Lesson 2.1 · Study time: 75 minutes · Language: English

Learning objectives

By the end of this lesson you will be able to:

Describe the main frameworks used to structure professional analysis of multi-agent and agentic ai workflows.
Select an appropriate framework for a given problem and justify the choice against alternatives.
Apply The transformer architecture to a real case and produce a defensible output.
Recognise the failure modes of framework-driven work, including false rigour and premature closure.

Key terms

The transformer architecture: Uses self-attention so every token can be weighted against every other, which is what makes long-context modelling practical.
Retrieval-augmented generation: Grounds a generative model in an external corpus so answers can be traced to retrievable passages.
The NIST AI Risk Management Framework: Organises practice into govern, map, measure and manage.
Model cards and datasheets: Document intended use, training data, evaluation results and known limitations.
Human-in-the-loop control: Places a named person at the point where an automated output becomes a consequential decision.

Lesson

What a framework is for

This lesson. Analytical Frameworks and Standards belongs to Module 2 — Frameworks and Standards of AI-A03 Multi-Agent and Agentic AI Workflows, taught in the School of Applied Artificial Intelligence. Work through the written lesson first, then the slide deck below, then the activity. Every element on this page is live HTML text: it can be enlarged, printed, searched, read aloud by assistive technology and machine-translated without loss.

A framework is a disciplined way of not fooling yourself. It does not generate conclusions; it forces you to make your reasoning visible so that someone else can find the weak joint. In applied artificial intelligence, the value of The transformer architecture is not that it produces a better answer than an expert’s intuition, but that it produces an answer whose basis can be inspected, challenged and revised when new evidence arrives. Intuition, however good, leaves no audit trail, and in multi-agent and agentic ai workflows an unauditable judgement is close to useless once it is contested.

This also explains why frameworks are unpopular with confident practitioners. They are slower, they surface disagreement that could have been left buried, and they often end by widening rather than narrowing the range of defensible conclusions. All three of those properties are features. A method that always confirms what the senior person in the room already thought is not a method; it is a ritual.

The main frameworks in this field

Five frameworks recur in professional work on multi-agent and agentic ai workflows. The transformer architecture uses self-attention so every token can be weighted against every other, which is what makes long-context modelling practical. Retrieval-augmented generation grounds a generative model in an external corpus so answers can be traced to retrievable passages. The NIST AI Risk Management Framework organises practice into govern, map, measure and manage. Model cards and datasheets document intended use, training data, evaluation results and known limitations. Finally, Human-in-the-loop control places a named person at the point where an automated output becomes a consequential decision. Each of these has a natural domain and a boundary beyond which it produces confident nonsense.

Choosing between them is a real analytical decision and should be recorded as one. The relevant questions are: what kind of uncertainty dominates this problem, how much time do I have, who has to be persuaded, and what will happen if I am wrong? A framework that is excellent for a six-month research project may be unusable in a forty-minute decision meeting, and the professional skill is to run a compressed but honest version rather than to abandon structure altogether.

Applying a framework properly

Take The transformer architecture. Applied properly it has four steps. First, state the question in a form that has more than one possible answer — questions with only one admissible answer are advocacy, not analysis. Second, lay out the alternatives explicitly, including the one you find least attractive, and give each a fair statement in its own strongest terms. Third, list the evidence and note against each item which alternatives it supports, which it weakens and, crucially, which it does not discriminate between at all. Fourth, write the conclusion together with the specific evidence that would overturn it.

The fourth step is the one most often skipped and the one that carries most of the value. An analysis that cannot say what would change its mind is not falsifiable and therefore cannot be tested by events. Recording the falsifier also converts your product into a monitoring instrument: if the named condition appears, the conclusion is due for revision, and someone can be assigned to watch for it.

How frameworks fail

The first failure is false rigour, in which the apparatus of a method is displayed while the judgements inside it remain unexamined. A neatly formatted matrix populated with guesses is not more reliable than the guesses; it is less reliable, because the format signals a confidence the content does not support. The second failure is premature closure, in which the framework is used to justify a conclusion that was reached before the work began. The tell is that the alternatives are stated weakly and the discriminating evidence all happens to point one way.

The third failure is scope error — applying a framework outside the conditions it assumes. The NIST AI Risk Management Framework, for example, is built for situations in which the relevant categories are reasonably stable; used in a fast-moving situation it will impose an order that is not there. The remedy in every case is the same and is unglamorous: state your assumptions before you start, name the conditions under which the framework would not apply, and check at the end whether those conditions held.

Worked example: running The transformer architecture under time pressure

A colleague asks for a view on multi-agent and agentic ai workflows by the end of the day. There is no time for a full application, so run the compressed version. Spend ten minutes writing the question and three genuine alternatives. Spend twenty minutes listing every piece of evidence you already have and marking, for each, whether it discriminates between the alternatives or is consistent with all of them. In practice a large share of the available evidence turns out to be non-discriminating, and simply seeing that is often the most useful thing you can tell your colleague.

Spend the final ten minutes writing three sentences: the judgement, the confidence level with the reason for it, and the one development that would change the assessment. That output is short enough to be read, honest about its limits, and testable against what happens next. It is also markedly better than the alternative product of the same forty minutes, which is a fluent paragraph asserting the first plausible answer with unearned certainty.

Slide deck

A ten-slide presentation of this lesson, built in HTML so that it prints, translates, enlarges and reads aloud on any device.

Slide 1 of 10