AI-A02 — APIs, Retrieval-Augmented Generation, and Knowledge Systems

Course Code: AI-A02  |  School: School of Artificial Intelligence  |  Cluster: Level 5 — Advanced Systems & Innovation

Level: Advanced  |  Duration: 6 weeks · 24–30 learning hours  |  Language: English  |  Certificate: Professional Certificate (non-degree)  |  Format: Self-paced with AI support under human supervision

Overview

A language model with no access to your institution’s documents can only tell you what is broadly true of the world. Making it useful for institutional work means grounding its answers in sources you control, and that is an information retrieval problem long before it is a model problem. This course is organised on that premise.

Learners build a working retrieval-grounded question answering system over a real corpus, and — more importantly — build the evaluation harness that tells them whether it works. Considerable time is spent on the parts practitioners skip: document preparation and chunking, metadata design, the difference between lexical and dense retrieval and why hybrid approaches usually win, and the measurement of retrieval quality separately from generation quality. A system that answers well because the generator guessed correctly is not a system that will keep working.

The course also treats API integration properly, since retrieval systems live inside pipelines. Authentication, rate limits, idempotency, error handling and cost control are covered as engineering concerns with institutional consequences, and API security is treated as a first-class topic rather than an afterthought.

Learning outcomes

On completion, a successful learner will be able to:

1. Prepare a document corpus for retrieval, with a chunking and metadata strategy justified against the query patterns you expect.
2. Explain and compare lexical and dense retrieval, and implement a hybrid approach with reranking.
3. Build a retrieval-grounded question answering pipeline that cites its source passages.
4. Evaluate retrieval and generation separately, using a test set with known answers and appropriate metrics.
5. Integrate external services by API with correct authentication, rate-limit handling, retries and idempotency.
6. Assess and mitigate the security exposures of an API-connected retrieval system, including injection through retrieved content.
7. Control and forecast the running cost of the system under realistic query volume.

Who this course is for

Technical staff and technically confident professionals building institutional knowledge systems: data and information officers, research engineers, library systems staff, and consultants implementing retrieval solutions.

Prerequisites

AI-F03 and AI-A01, or equivalent. Comfort with basic scripting in Python or an equivalent language, reading JSON, and using a command line. This is the most technical course in the school and learners without scripting ability should take AI-P07 instead.

Syllabus

Module 1 — Retrieval before generation

Focus. Framing the problem as information retrieval. Query patterns, relevance, and the recognition that generation quality is bounded above by retrieval quality.

Lessons. 1.1 Why this is a retrieval problem. 1.2 Query patterns and what users actually ask. 1.3 Relevance and its measurement. 1.4 The bound that retrieval places on generation.

Core reading. Christopher D. Manning, Prabhakar Raghavan & Hinrich Schütze, Introduction to Information Retrieval (Cambridge: Cambridge University Press, 2008), chapters 1 and 8.

Deliverable. Query analysis for a real corpus, with at least thirty representative questions classified by type.

Module 2 — Corpus preparation, chunking and metadata

Focus. The unglamorous determinant of system quality. Document conversion and cleaning, chunk size and overlap, structural chunking versus fixed windows, and metadata that supports filtering, currency and permission.

Lessons. 2.1 Conversion, cleaning and what gets lost. 2.2 Chunk size, overlap and structure. 2.3 Metadata schema design. 2.4 Currency, authority and permission fields.

Core reading. Timnit Gebru et al., “Datasheets for Datasets”, Communications of the ACM 64, no. 12 (2021), applied to corpus documentation.

Deliverable. Prepared corpus with documented chunking strategy and metadata schema.

Module 3 — Lexical, dense and hybrid retrieval

Focus. How BM25-family scoring works and why it remains strong; how dense retrieval differs and where it wins; reranking; and the empirical case for hybrid retrieval in institutional corpora with specialised vocabulary.

Lessons. 3.1 Lexical scoring and the probabilistic relevance framework. 3.2 Dense passage retrieval. 3.3 Hybrid retrieval and fusion. 3.4 Reranking and its cost.

Core reading. Stephen Robertson & Hugo Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, Foundations and Trends in Information Retrieval 3, no. 4 (2009): 333–389. Vladimir Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering”, Proceedings of EMNLP (ACL, 2020).

Deliverable. Retrieval comparison report across lexical, dense and hybrid configurations on your corpus.

Module 4 — Grounded generation and citation

Focus. Assembling the prompt from retrieved passages, constraining the generator to the supplied context, requiring citation, and handling the case where the corpus does not contain the answer.

Lessons. 4.1 Context assembly and ordering effects. 4.2 Constraining to supplied context. 4.3 Citation to passage, not to document. 4.4 Refusal when the answer is absent.

Core reading. Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 33 (2020). Ziwei Ji et al., “Survey of Hallucination in Natural Language Generation”, ACM Computing Surveys 55, no. 12 (2023).

Deliverable. Working pipeline with passage-level citation and a documented refusal behaviour.

Module 5 — Evaluation as an engineering discipline

Focus. Separating retrieval evaluation from generation evaluation. Building a labelled test set, measuring recall and precision at k, and assessing groundedness and citation accuracy. Regression testing when the corpus or model changes.

Lessons. 5.1 Building a labelled test set. 5.2 Retrieval metrics at k. 5.3 Groundedness and citation accuracy. 5.4 Regression testing and version pinning.

Core reading. Manning, Raghavan & Schütze, Introduction to Information Retrieval, chapter 8 on evaluation. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 28 (2015).

Deliverable. Evaluation harness with results, including at least ten adversarial and out-of-scope queries.

Module 6 — APIs, security and cost

Focus. Integrating the system into a pipeline safely. Authentication and secret handling, rate limits, retries and idempotency, injection through retrieved content, permission enforcement, and cost forecasting under real load.

Lessons. 6.1 Authentication, secrets and least privilege. 6.2 Rate limits, retries, idempotency. 6.3 Indirect prompt injection through the corpus. 6.4 Permission enforcement and cost forecasting.

Core reading. OWASP, API Security Top 10 and Top 10 for Large Language Model Applications (OWASP Foundation, current editions). Leonard Richardson & Sam Ruby, RESTful Web Services (Sebastopol: O’Reilly Media, 2007), chapters 1–4.

Deliverable. Final submission: deployed pipeline, evaluation report, security assessment and cost model.

Assessment

Pass mark 70 per cent. All assessed components must be attempted. Every mark in this course is issued by a human assessor; no assessment outcome is generated automatically.

Rubric criteria

Each assessed artefact is marked against four criteria at four levels (distinction, pass with merit, pass, fail).

1. Measurement discipline: are retrieval and generation evaluated separately, with metrics appropriate to each?
2. Engineering quality: does the pipeline handle failure, permission and cost rather than only the happy path?
3. Grounding integrity: does every answer cite a retrieved passage, and does the system refuse when the corpus is silent?
4. Security posture: are injection through retrieved content and permission bypass addressed with specific controls?

Reading list

Core. Christopher D. Manning, Prabhakar Raghavan & Hinrich Schütze, Introduction to Information Retrieval (Cambridge: Cambridge University Press, 2008).

Peer-reviewed. Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 33 (2020). Vladimir Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering”, EMNLP (ACL, 2020). Stephen Robertson & Hugo Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, Foundations and Trends in Information Retrieval 3, no. 4 (2009). Ziwei Ji et al., “Survey of Hallucination in Natural Language Generation”, ACM Computing Surveys 55, no. 12 (2023). D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 28 (2015).

Engineering and security. Leonard Richardson & Sam Ruby, RESTful Web Services (Sebastopol: O’Reilly Media, 2007). OWASP, API Security Top 10 and Top 10 for Large Language Model Applications (current editions). Timnit Gebru et al., “Datasheets for Datasets”, Communications of the ACM 64, no. 12 (2021).

All items are published works identifiable by author, title and publisher. Learners obtain them through an institutional library or the publisher. The Academy does not distribute copyrighted texts.

Academic integrity and use of AI

Generative tools may be used in producing assessed work under three conditions. Use must be disclosed in a short statement appended to each submission, naming the tool and the task it performed. Any factual or technical claim originating from a generative tool must be verified against a citable source before it enters assessed work, and the verification must be evidenced. The analytical judgement in each artefact must be the learner’s own and must be defensible in a short follow-up. Corpora must be material the learner is authorised to process. API keys and secrets must never appear in submitted work; submissions containing live credentials are deleted and must be resubmitted.

Instructor: pending owner confirmation. Pricing: pending owner approval. Reference list verified against publisher records; any later addition is marked for verification before publication.