HE-Q04 — Data and Analytics for Quality

Course Code: HE-Q04  |  School: School of Higher Education  |  Cluster: Quality Assurance

Level: Intermediate  |  Estimated duration: 6 weeks (approximately 40 hours)  |  Language: English  |  Certificate: Professional Certificate (non-degree)  |  Format: Self-paced with AI support under human supervision

Overview

Universities generate more data than they use and use more data than they understand. Student record systems, virtual learning environments, library systems, finance and human resources each produce continuous streams of information, and quality processes increasingly demand that decisions be evidenced by them. This course develops the capability to turn that raw material into defensible institutional judgement.

The course is built around three linked competencies: constructing indicators that measure what they claim to measure, analysing institutional data with appropriate statistical caution, and presenting results so that non-specialist decision-makers reach accurate conclusions rather than convenient ones. Considerable attention is given to the standard pathologies — comparing non-comparable cohorts, mistaking small-sample noise for trend, and treating survey satisfaction as a proxy for educational quality.

The final third of the course addresses learning analytics and predictive modelling directly, including their ethical structure. Predictive systems that flag students as at risk change how those students are treated and how they see themselves, and the literature on algorithmic bias and on the obligation to act after prediction is now substantial enough to constitute required reading for anyone deploying such a system.

Learning outcomes

On completion, a successful learner will be able to:

1. Map the data assets of a higher education institution and assess their quality, coverage and interoperability.
2. Construct and validate quality indicators, stating explicitly what each does and does not measure.
3. Apply appropriate descriptive and inferential techniques to institutional datasets and recognise when a technique is inappropriate.
4. Interpret longitudinal and cohort data, distinguishing genuine trend from compositional change and random variation.
5. Design data visualisations and dashboards that support accurate rather than merely rapid interpretation.
6. Evaluate learning analytics and predictive models for validity, bias, and the ethical obligations they create.
7. Produce an analytical report that informs a specific institutional quality decision.

Who this course is for

Institutional researchers, planning and analytics staff, quality officers, registry personnel, and academic leaders who commission or consume institutional data. No programming background is assumed.

Prerequisites

Comfort with spreadsheets and basic quantitative reasoning. HE-Q01 Quality Assurance in Higher Education is recommended for context. Statistical methods are introduced from first principles at the level required.

Syllabus

Module 1 — The Institutional Data Estate

Before analysis is possible the institution’s data must be located, described and judged for fitness. This module conducts that audit and surfaces the definitional inconsistencies that silently corrupt most institutional reporting.

Lessons. 1.1 Student records, VLE, library, finance and HR data sources  ·  1.2 Definitional inconsistency: what counts as an enrolled student  ·  1.3 Data quality dimensions: accuracy, completeness, timeliness, consistency  ·  1.4 Governance, stewardship and access control

Deliverable. A diagnostic note auditing three institutional data sources and documenting at least four definitional inconsistencies between them.

Module 2 — Constructing and Validating Indicators

An indicator is a theory about what matters compressed into a number, and most institutional indicators are never tested against that theory. This module builds indicators deliberately and validates them.

Lessons. 2.1 From construct to operational definition  ·  2.2 Composite indicators, weighting and the arbitrariness problem  ·  2.3 Construct, criterion and face validity in institutional measurement  ·  2.4 Documenting an indicator so others compute it identically

Deliverable. An analytical brief specifying three quality indicators with full operational definitions, validity arguments and stated limitations.

Module 3 — Analysing Institutional Data

Most institutional analysis requires only a small set of techniques applied carefully; the failures arise from careless application rather than missing sophistication. This module builds that care.

Lessons. 3.1 Descriptive statistics, distributions and the tyranny of the mean  ·  3.2 Cohort tracking, progression rates and survival curves  ·  3.3 Comparison, disaggregation and Simpson’s paradox  ·  3.4 Uncertainty, small numbers and when not to report

Deliverable. A design artefact: a fully documented analysis of one institutional dataset, including an uncertainty statement and a list of analyses deliberately not performed.

Module 4 — Visualisation and Decision Support

A dashboard is an argument about what deserves attention, and poorly designed dashboards reliably produce confident wrong decisions. The module applies established visualisation principles to institutional reporting.

Lessons. 4.1 Principles of graphical integrity and data-ink economy  ·  4.2 Chart selection and the misuse of dual axes and truncated scales  ·  4.3 Dashboard design for governance versus operational use  ·  4.4 Narrative structure in analytical reporting

Deliverable. An evaluation report critiquing an existing institutional dashboard against graphical integrity principles, with an annotated redesign.

Module 5 — Learning Analytics and Its Ethics

Predictive analytics in education raises questions that are not answered by predictive accuracy: what the institution owes a student it has flagged, and what happens when the flag becomes a label. This module treats those as design questions.

Lessons. 5.1 Learning analytics: sources, models and demonstrated effects  ·  5.2 Algorithmic bias and differential validity across student groups  ·  5.3 The obligation to act and the ethics of knowing  ·  5.4 Consent, transparency and student agency over their own data

Deliverable. An implementation plan for a learning analytics deployment including a bias assessment, an intervention obligation, and a student transparency mechanism.

Module 6 — Capstone: An Institutional Analytics Report

The capstone answers a real institutional quality question with real or realistically constructed data, and defends every methodological choice made. It is assessed as a document that could be tabled at a quality committee.

Lessons. 6.1 Framing an answerable question  ·  6.2 Selecting data and declaring limitations  ·  6.3 Analysis and visualisation  ·  6.4 Recommendations and their evidentiary basis

Deliverable. A 2,500-word analytics report answering one institutional quality question, with methodology, visualisations, limitations and recommendations.

Assessment

Pass mark 70 per cent. All assessed components must be attempted. Every mark in this course is issued by a human assessor; no assessment outcome is generated automatically.

Rubric criteria

Each assessed artefact is marked against four criteria at four levels (distinction, pass with merit, pass, fail).

1. Methodological soundness. Techniques are appropriate, assumptions are stated, and uncertainty is quantified or acknowledged.
2. Measurement validity. Indicators are properly operationalised and their limits are declared rather than concealed.
3. Communicative integrity. Visualisations and narrative lead a reasonable reader to accurate conclusions.
4. Ethical reasoning. Bias, consent, labelling and the obligation to act are addressed substantively.

Reading list

Core. Howard, R. D., McLaughlin, G. W. and Knight, W. E. (eds.) (2012) The Handbook of Institutional Research. San Francisco: Jossey-Bass.

• Siemens, G. and Long, P. (2011) ‘Penetrating the Fog: Analytics in Learning and Education’, EDUCAUSE Review, 46(5), pp. 30–40.
• Prinsloo, P. and Slade, S. (2017) ‘An Elephant in the Learning Analytics Room: The Obligation to Act’, in Proceedings of the Seventh International Learning Analytics and Knowledge Conference. New York: ACM, pp. 46–55.
• O’Neil, C. (2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York: Crown.
• Tufte, E. R. (2001) The Visual Display of Quantitative Information, 2nd edn. Cheshire, CT: Graphics Press.
• Muller, J. Z. (2018) The Tyranny of Metrics. Princeton: Princeton University Press.

Academic integrity and use of AI

Generative AI may be used in this course as a drafting, translation and critique aid, and its use must be disclosed. Every submission carries a short use-of-AI statement naming the tools used, the tasks they performed and the checks the learner applied to the output. Using AI to fabricate data, invent sources, impersonate an interview participant, or produce an artefact the learner cannot explain in a live viva is prohibited.

All citations are verified before submission; a reference that cannot be located by the assessor is treated as fabricated. Fabrication, plagiarism and undisclosed ghost-authorship result in failure of the component and referral to the academic integrity panel. Marks in this course are issued by human assessors, and any learner may be asked to defend a submission orally before a mark is confirmed.