D-optimal adaptive comparative judgment

Abstract

Comparative judgment has recently been explored as an alternative to traditional marking in education, offering advantages in certain contexts. Yet, the sheer number of possible script pairs makes full pairwise comparison impractical. Existing adaptive algorithms often lack a formal foundation, raising concerns about fairness in assessment.

To address this, we examine comparative judgment through the Bradley–Terry model and propose an adaptive optimal design for paired comparisons. Drawing on experimental design theory, we develop efficient methods for selecting sets of pairs, adapting the multiple incomplete cyclic design and general equivalence theorem to ensure D‑optimality. This framework provides a principled approach for constructing and validating adaptive comparative judgment.

We evaluate the method using simulated data, comparing it against full factorial paired comparison. Results reveal similar patterns in perceived script quality across both approaches, but the adaptive method requires far fewer comparisons, offering a more efficient and equitable solution for educational assessment.

Keywords

• Assessment design
• Comparative judgement
• Fairness
• Marking
• Reliability
• Statistics