AI-A04 — AI Product and Prototype Development

Course Code: AI-A04  |  School: School of Artificial Intelligence  |  Cluster: Level 5 — Advanced Systems & Innovation

Level: Advanced  |  Duration: 6 weeks · 24–30 learning hours  |  Language: English  |  Certificate: Professional Certificate (non-degree)  |  Format: Self-paced with AI support under human supervision

Overview

The failure rate of AI products is not primarily a modelling failure. Systems fail because they solve a problem nobody had, because the interface gives users no way to tell a good answer from a plausible one, because the economics collapse at scale, or because the organisation that commissioned the product cannot absorb what it produces. This course is built on the premise that the scarce skill is not building with a model but deciding what to build, proving it is worth building, and shipping something that a real user can trust.

The course follows a full development arc compressed into six weeks: discovery and problem framing, opportunity assessment, interaction design for probabilistic systems, prototype construction, evaluation with real users, and the business and governance case. It draws on the established discipline of product management — continuous discovery, opportunity solution trees, evidence-based prioritisation — and adapts it to the particular properties of AI systems, which are non-deterministic, expensive per unit of use, and capable of being confidently wrong.

Design is treated as a first-order engineering concern rather than decoration. A substantial part of the course is devoted to the interaction patterns that make probabilistic output usable: setting expectations before first use, communicating confidence honestly, exposing sources, designing graceful failure, and building correction paths that feed back into improvement. Learners finish with a working prototype, evidence from real users, a unit-economics model and a launch recommendation they can defend.

Learning outcomes

On completion, a successful learner will be able to:

1. Frame a product opportunity as a specific user problem with evidence of its frequency, cost and current workaround, and state the falsifiable assumption on which the proposed solution depends.
2. Conduct structured discovery interviews and synthesise findings into an opportunity solution tree that links desired outcomes to candidate solutions and to the experiments that would test them.
3. Assess whether a problem warrants an AI solution at all, using criteria of tolerance for error, availability of ground truth, cost of verification and value of partial correctness.
4. Design interaction patterns appropriate to probabilistic systems, including expectation setting, confidence communication, source attribution, graceful failure and user correction.
5. Build a functioning prototype of an AI-enabled product sufficient for real users to complete a genuine task unaided.
6. Plan and run an evaluation with real users combining task-completion measures, error and recovery observation, and structured qualitative feedback, and report the results without advocacy.
7. Construct a unit-economics model covering inference cost, verification labour, support burden and expected value, and produce a launch, iterate or stop recommendation grounded in it.

Who this course is for

The course is intended for people who must turn AI capability into something an institution can actually use: product and service managers in universities, publishers, ministries and NGOs; innovation and digital transformation staff; researchers commercialising a method; founders at the prototype stage; and technical staff who have built demonstrations that never became products. It suits participants who can commit to speaking with real users during the course, since the assessment depends on evidence gathered from outside the classroom.

Prerequisites

Completion of AI-A01 or AI-A02, or equivalent practical experience building with language models or retrieval systems. Learners should be able to assemble a working prototype using either code or a low-code platform, and must have access to at least five potential users of the product they intend to develop. No prior product management training is assumed.

Syllabus

Module 1 — Problem framing and the discipline of not building

The module begins with the most expensive error in the field: building a technically impressive system for a problem that does not merit it. It covers problem statements that are specific enough to be wrong, the distinction between a user outcome and a feature request, and the practice of documenting the current workaround, because a problem with no existing workaround is usually not a problem.

Lessons. 1.1 Outcomes versus features. 1.2 Writing a falsifiable problem statement. 1.3 Mapping the current workaround and its true cost. 1.4 The criteria for when AI is the wrong instrument.

Learners are required to articulate the condition under which they would abandon the product. This is revisited in the final module and marked against the evidence they subsequently gathered.

Deliverable. Problem statement, workaround map and stated abandonment condition (two pages).

Module 2 — Discovery: interviewing, synthesis and the opportunity tree

This module teaches continuous discovery as a habit rather than a phase. It covers interview technique that elicits past behaviour instead of speculative preference, the sampling of users who are not enthusiasts, synthesis into opportunities, and the construction of an opportunity solution tree connecting outcome, opportunity, solution and test.

Lessons. 2.1 Interviewing for behaviour, not opinion. 2.2 Recruiting sceptics and non-adopters. 2.3 Synthesis and opportunity mapping. 2.4 Choosing the smallest experiment that could disconfirm your assumption.

Learners conduct a minimum of five interviews and are assessed on the quality of their synthesis rather than on whether the findings support their original idea. Evidence that contradicts the initial hypothesis is explicitly rewarded.

Deliverable. Discovery report with interview evidence and an opportunity solution tree.

Module 3 — Designing for probabilistic systems

Conventional interface conventions assume deterministic software: the same action produces the same result, and errors are the exception. AI products violate both assumptions. This module works through the design patterns that address this — onboarding that calibrates expectation, output presented as a draft rather than an answer, visible sources, uncertainty expressed without false precision, and failure states that leave the user with a path forward.

Lessons. 3.1 Calibrating expectation before first use. 3.2 Communicating confidence and provenance. 3.3 Designing the correction path. 3.4 Graceful failure and the cost of automation bias.

The module draws on established human–AI interaction guidance and examines automation bias directly: interfaces that present output with unwarranted authority reliably degrade the quality of human judgement, and this is treated as a design defect rather than a user failing.

Deliverable. Interaction specification with annotated wireframes covering the success, uncertain and failure states.

Module 4 — Building the prototype

The build module covers scoping a prototype to the single riskiest assumption, selecting the thinnest technical stack that permits a real user to complete a real task, instrumentation for later evaluation, and the discipline of shipping something unfinished. It addresses the specific engineering concerns of AI prototypes: latency perception, streaming, caching, fallback behaviour when the model is unavailable, and cost control during testing.

Lessons. 4.1 Scoping to the riskiest assumption. 4.2 Thin stacks and disposable code. 4.3 Instrumentation and event logging. 4.4 Latency, caching, fallback and cost control.

Prototypes are expected to be genuinely functional for one narrow path. Breadth is penalised; a system that does one task reliably outscores one that gestures at five.

Deliverable. Working prototype with instrumentation and a scoping note explaining what was deliberately left out.

Module 5 — Evaluation with real users

This module runs the prototype against real users and treats the result as evidence rather than validation. It covers protocol design, task selection, the observation of error and recovery, the separation of what users say from what they do, and analysis that reports failure prominently. It also covers evaluation of the AI component itself: a small golden set, a measure of output quality and a record of the cases where the system failed silently.

Lessons. 5.1 Designing an honest evaluation protocol. 5.2 Observing error, recovery and abandonment. 5.3 Golden sets and output quality measurement. 5.4 Analysis and reporting without advocacy.

The characteristic error at this stage is the demonstration disguised as a test. Learners are assessed on whether their protocol could have produced a negative result and on how they responded to the negative results they obtained.

Deliverable. Evaluation report with protocol, results, observed failures and revised assumptions.

Module 6 — Economics, governance and the launch decision

The final module converts the work into a decision. It covers unit economics for AI products — inference and retrieval cost per task, the labour cost of human verification, support and remediation burden, and the value of the outcome delivered — and the sensitivity of that model to volume and model pricing. It then addresses the governance requirements of shipping: data handling, retention, disclosure to users that they are interacting with an AI system, accessibility, and the named owner of the system in production.

Lessons. 6.1 Unit economics and sensitivity analysis. 6.2 Data protection, retention and disclosure obligations. 6.3 Accessibility and equitable performance across user groups. 6.4 Writing the launch, iterate or stop recommendation.

The course ends with the recommendation itself, presented as a document a director could sign, including residual risk and the conditions under which the decision should be revisited.

Deliverable. Final project: prototype, evaluation report, unit-economics model, governance note and launch recommendation.

Assessment

Pass mark 70 per cent. All assessed components must be attempted. Every mark in this course is issued by a human assessor; no assessment outcome is generated automatically.

Rubric criteria

Each assessed artefact is marked against four criteria at four levels (distinction, pass with merit, pass, fail).

1. Evidential grounding. Whether claims about user need rest on documented behaviour gathered from real people, whether disconfirming evidence was sought, and whether the learner changed position when the evidence required it.
2. Design quality for uncertainty. Whether the interface sets accurate expectations, communicates provenance and confidence without false precision, and provides a usable path out of error.
3. Technical execution. Whether the prototype functions reliably on its intended path, is instrumented well enough to be evaluated, and handles latency, failure and cost in a way that would survive real use.
4. Commercial and institutional judgement. Whether the economics are modelled with defensible assumptions, whether governance obligations are identified and addressed, and whether the final recommendation follows from the evidence rather than from prior commitment.

Reading list

Core. Marty Cagan, Inspired: How to Create Tech Products Customers Love, 2nd edition (Hoboken: John Wiley & Sons, 2018).

Teresa Torres, Continuous Discovery Habits: Discover Products that Create Customer Value and Business Value (Product Talk LLC, 2021).

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu et al., ‘Guidelines for Human-AI Interaction’, in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (New York: ACM, 2019).

Google PAIR, People + AI Guidebook (Mountain View: Google, 2019; revised edition).

Erik Brynjolfsson, ‘The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligence’, Daedalus, 151:2 (2022), pp. 272–287.

National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (Gaithersburg: NIST, 2023).

Academic integrity and use of AI

Learners are expected to build with AI tools throughout this course, and to use them in drafting where it is useful. Every submission must carry a use-of-AI statement naming the tools used, the components they contributed to, and the changes the learner made after review. The evidence gathered from users, however, must be genuine: fabricated interviews, invented quotations and synthetic user data presented as real are treated as the most serious form of academic misconduct in this course, because the entire discipline being taught depends on the integrity of that evidence.

Participants are responsible for the lawful and ethical treatment of the people they interview and test with. Informed consent must be obtained and recorded, participants must be able to withdraw, and personal data must be minimised, anonymised in submissions, and never placed into a third-party system without a lawful basis and the participant’s knowledge.