Most chosen general qualifications exam board in England.

  • About AQA

  • Centre Services

  • Join Us

  • Contact Us

AQA
  • Subjects
  • Qualifications
  • Professional Development
  • Exams Admin
  • Services
  • Search
  • Subjects

  • Qualifications

  • Professional Development

  • Exams Admin

  • Services

  • About AQA

  • Centre Services

  • Join Us

  • Contact Us

  • Log in

Subjects

  • Accounting

  • Art and Design

  • Biology

  • Business

  • Chemistry

  • Computer Science

  • Dance

  • Design and Technology

  • Drama

  • Economics

  • English

  • Food preparation and Nutrition

  • French

  • Geography

  • German

  • History

  • Law

  • Mathematics

  • Media Studies

  • Music

  • Physical Education

  • Physics

  • Politics

  • Psychology

  • Religious Studies

  • Science

  • Sociology

  • Spanish

  • All subjects

GCSEs

  • Biology (8461)

  • Chemistry (8462)

  • Combined Science: Trilogy (8464)

  • English Language (8700)

  • English Literature (8702)

  • Geography (8035)

  • History (8145)

  • Mathematics (8300)

  • See all GCSEs

AS and A-levels

  • Biology (7401)

  • Business (7131)

  • Chemistry (7404)

  • Geography (7037)

  • History (7041)

  • Physics (7407)

  • Psychology (7181)

  • Sociology (7191)

  • See all AS and A-Levels

Other qualifications

  • Applied Generals

  • AQA Certificate Mathematics

  • Entry Level Certificates

  • Project Qualifications

  • Unit Award Scheme

  • All qualifications

Our training

  • Course finder

  • About our training

  • Online training

  • Face-to-face training

  • In-school training

  • Inside assessment

Courses by theme

  • Effective exam prep

  • Exams officers

  • Getting started

  • Unit Award Scheme

Courses by subject

  • English

  • Mathematics

  • Science

  • Languages

  • Design and Technology

  • Physical Education

  • Geography

  • History

  • All professional development

Dates

  • Dates and timetables

  • Key dates

Non-exam assessment (NEA)

  • NEA, coursework and controlled assessment

  • Deadlines for non-exam assessment

  • Record forms

  • Submit marks

Exams

  • Entries

  • Entry fees

  • Exams guidance

  • Question papers and stationery

  • Access arrangements

  • Special consideration

Results

  • Results days

  • Results slips

  • Grade boundaries

  • Results statistics

  • Post-results services

  • Exam certificates

  • All Exams Admin

Assessment Services

  • Centre Services

  • Associate Extranet

  • Become an associate

Products

  • All About Maths

  • Alpha Plus

  • Data Insights

  • Exampro

  • Project Q

  • Stride Maths

  • Testbase

  • Unit Award Scheme

News and Insights

  • AQI research and insight

  • News

  • Inside exams podcast

AQA
  • Become an examiner
  • Switch to AQA
  • Contact Us
  • Join us
  • Terms and conditions
  • Accessibility
  • Modern slavery statement
  • Privacy notice
  • Cookie notice
  • X
  • LinkedIn
  • Youtube

©AQA 2025 | Company number: 03644723 | Registered office: Devas Street, Manchester, M15 6EX | AQA is not responsible for the content of external sites

AQA Education has obtained an injunction preventing interference with public examinations. This notice is to alert you to the injunction, so that you are aware of it and can make submissions about it if you wish to do so.

  1. Home
  2. About Us
  3. Our Research
  4. Research Library
  5. Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment

Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment

Share this page

  • WhatsApp
  • LinkedIn
  • X
  • Facebook
Using adaptive comparative judgement to obtain a highly reliable rank order in
            summative assessment

Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment

20 Jun 2012

PDF | 1.61 MB

Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment

By Claire Whitehouse, Alastair Pollitt

Abstract

Adaptive Comparative Judgment (ACJ) offers an alternative to marking, especially for performance assessments for which achievement can be difficult to describe in mark schemes. The ACJ system uses a web-browser to deliver pairs of candidates’ work to judges. The judges apply their subject expertise to select which of the two sets of work is better in terms of meeting appropriate educational objectives.

A quality parameter is estimated for each piece of candidates’ work using the Rasch logistic model to analyse the judges’ decisions. Placing the pieces of work on a measurement scale gives them a rank order. The adaptive nature of ACJ lies in the pairing of essays for each judge. The selection of essays is based on a maximum distance between the quality parameter estimates, allowing useful information to be gained from each paired comparison.

This paper reports a study in which ACJ was used to rank order a random sample of 564 essays on a topic in physical geography based on the judgements of a group of 23 teachers and examiners. The reliability of the rank order was 0.97. Evidence is presented for the need for judges to be teaching at the same qualification level as they are judging at. There is a discussion of the factors that need to be addressed before implementing ACJ in summative assessment.

How to cite

Whitehouse, C. and Pollitt, A (2012). Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment. Manchester: AQA Centre for Education Research and Policy.

Keywords

  • Comparative judgement
  • Reliability