HE-Q05 — Benchmarking and Rankings

Course Code: HE-Q05  |  School: School of Higher Education  |  Cluster: Quality Assurance

Level: Intermediate  |  Estimated duration: 6 weeks (approximately 40 hours)  |  Language: English  |  Certificate: Professional Certificate (non-degree)  |  Format: Self-paced with AI support under human supervision

Overview

Global university rankings are, by any reasonable methodological standard, weak instruments. They compress incommensurable activities into single scores, weight their components by editorial judgement rather than evidence, rely heavily on reputation surveys with low and unrepresentative response rates, and systematically favour old, wealthy, English-language, research-intensive institutions. They are also enormously consequential. Governments set policy by them, students choose by them, and vice-chancellors are dismissed over them.

This course refuses both available postures — uncritical adoption and dismissive contempt — and instead teaches learners to read rankings as artefacts: to open the methodology, identify the levers, and understand precisely what a movement of ten places does and does not indicate. Wendy Espeland and Michael Sauder’s ethnographic work on how rankings reshape the organisations they measure is central to the course.

The constructive half of the course is benchmarking, which is the disciplined alternative. Where rankings compare an institution against all others on someone else’s criteria, benchmarking compares it against deliberately chosen peers on criteria the institution has reasons to care about. Learners build a full benchmarking exercise: peer selection, metric definition, data collection, gap analysis and improvement targeting.

Learning outcomes

On completion, a successful learner will be able to:

1. Deconstruct the methodology of a major global ranking and identify its indicator weights, data sources and structural biases.
2. Explain the documented effects of rankings on institutional behaviour, resource allocation and academic work.
3. Assess ranking methodologies against the Berlin Principles and the Leiden Manifesto.
4. Select a defensible peer comparison group and justify the selection criteria.
5. Design a benchmarking framework with comparable metrics and a documented data collection protocol.
6. Conduct a gap analysis against peers and translate gaps into prioritised, feasible improvement targets.
7. Advise institutional leadership on ranking strategy in a way that distinguishes genuine improvement from position management.

Who this course is for

Planning and strategy staff, institutional researchers, communications and marketing leads with reputation responsibility, deans, and senior leaders accountable for institutional positioning. Also relevant to ministry officials who use rankings in policy.

Prerequisites

No formal prerequisites. HE-Q04 Data and Analytics for Quality is helpful for the metric construction work but the course is self-contained.

Syllabus

Module 1 — Anatomy of a Ranking

Rankings are constructed objects, and their construction is public. This module opens the methodology documents of the major global systems and reads them as one would read any measurement instrument.

Lessons. 1.1 Indicator sets and weights in the principal global rankings  ·  1.2 Reputation surveys: sampling, response rates and circularity  ·  1.3 Bibliometric indicators, field normalisation and language bias  ·  1.4 Reconstructing a ranking score from published components

Deliverable. A diagnostic note reconstructing one institution’s score in a major ranking and identifying the two indicators that dominate its position.

Module 2 — What Rankings Do to Institutions

Espeland and Sauder show that rankings do not merely describe institutions but reshape them, redirecting resources toward measured activity and generating organisational anxiety. This module examines that reactivity empirically.

Lessons. 2.1 Reactivity, self-fulfilling prophecy and commensuration  ·  2.2 Resource reallocation toward measured activities  ·  2.3 Effects on hiring, admissions and institutional mission  ·  2.4 Gaming, data manipulation and documented scandals

Deliverable. An analytical brief documenting three changes in one institution’s behaviour attributable to ranking pressure, with evidence.

Module 3 — Standards for Judging Rankings

The Berlin Principles and the Leiden Manifesto provide explicit criteria against which a ranking or metric system can be assessed. The module applies them systematically rather than rhetorically.

Lessons. 3.1 The Berlin Principles on Ranking of Higher Education Institutions  ·  3.2 The Leiden Manifesto for research metrics  ·  3.3 Transparency, auditability and methodological stability  ·  3.4 Assessing national and regional rankings against the same standards

Deliverable. A design artefact: a scored assessment of one ranking system against all Berlin Principles, with evidence for each judgement.

Module 4 — Benchmarking as the Disciplined Alternative

Benchmarking replaces universal comparison with purposeful comparison, and its value depends almost entirely on the honesty of peer selection. The module builds the framework properly.

Lessons. 4.1 Competitive, collaborative and aspirational benchmarking  ·  4.2 Peer selection: aspirational, comparable and honest comparison sets  ·  4.3 Defining metrics that are genuinely comparable across institutions  ·  4.4 Data collection protocols and reciprocal agreements

Deliverable. An evaluation report presenting a peer group selection with explicit criteria and a defence against the charge of flattering selection.

Module 5 — Gap Analysis and Improvement Targeting

A benchmarking exercise that ends in a comparison table has failed. This module converts gaps into prioritised targets constrained by resource reality.

Lessons. 5.1 Quantifying and prioritising performance gaps  ·  5.2 Distinguishing gaps that are closable from those that are structural  ·  5.3 Target setting, timelines and the danger of arbitrary ambition  ·  5.4 Monitoring and revising the benchmarking cycle

Deliverable. An implementation plan converting a benchmarking gap analysis into three prioritised, resourced improvement targets.

Module 6 — Capstone: Positioning and Improvement Strategy

The capstone produces a strategy that distinguishes actions that would genuinely improve the institution from actions that would only improve its position, and argues for a defensible balance between them.

Lessons. 6.1 Establishing the ranking and benchmarking baseline  ·  6.2 Separating substance from position management  ·  6.3 Building the improvement case  ·  6.4 Advising leadership candidly

Deliverable. A 2,500-word institutional positioning and improvement strategy for one named institution, including ranking analysis, peer benchmarking, gap analysis and prioritised targets.

Assessment

Pass mark 70 per cent. All assessed components must be attempted. Every mark in this course is issued by a human assessor; no assessment outcome is generated automatically.

Rubric criteria

Each assessed artefact is marked against four criteria at four levels (distinction, pass with merit, pass, fail).

1. Methodological scepticism. Ranking methodologies are analysed technically and their biases identified with evidence.
2. Comparative rigour. Peer selection and metric definitions are justified and genuinely comparable.
3. Strategic honesty. The distinction between real improvement and position management is maintained explicitly.
4. Feasibility. Targets and timelines are constrained by realistic resource and structural limits.

Reading list

Core. Hazelkorn, E. (2015) Rankings and the Reshaping of Higher Education: The Battle for World-Class Excellence, 2nd edn. Basingstoke: Palgrave Macmillan.

• Espeland, W. N. and Sauder, M. (2016) Engines of Anxiety: Academic Rankings, Reputation, and Accountability. New York: Russell Sage Foundation.
• Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. and Rafols, I. (2015) ‘Bibliometrics: The Leiden Manifesto for Research Metrics’, Nature, 520(7548), pp. 429–431.
• IREG Observatory on Academic Ranking and Excellence (2006) Berlin Principles on Ranking of Higher Education Institutions. Berlin: IREG.
• Marginson, S. (2007) ‘Global University Rankings: Implications in General and for Australia’, Journal of Higher Education Policy and Management, 29(2), pp. 131–142.
• Alstete, J. W. (1995) Benchmarking in Higher Education: Adapting Best Practices to Improve Quality. ASHE-ERIC Higher Education Report No. 5. Washington, DC: George Washington University.

Academic integrity and use of AI

Generative AI may be used in this course as a drafting, translation and critique aid, and its use must be disclosed. Every submission carries a short use-of-AI statement naming the tools used, the tasks they performed and the checks the learner applied to the output. Using AI to fabricate data, invent sources, impersonate an interview participant, or produce an artefact the learner cannot explain in a live viva is prohibited.

All citations are verified before submission; a reference that cannot be located by the assessor is treated as fabricated. Fabrication, plagiarism and undisclosed ghost-authorship result in failure of the component and referral to the academic integrity panel. Marks in this course are issued by human assessors, and any learner may be asked to defend a submission orally before a mark is confirmed.