Statistical robustness in comparability studies: the choice of model and data selection

Abstract

Over the past few years, logistic multilevel models have been used to model the probability of candidates exceeding a given grade boundary in a given subject dependent upon the awarding body of entry. Questions remain over the validity and robustness of such an approach, not simply from a modelling point of view but also from a data selection point of view. Many of these questions were discussed at a seminar addressing the methodologies applied in recent comparability studies (Fowles, 2000) but, while differing views were freely expressed, there was little opportunity to consider any supporting statistical evidence.

This report collects together research from the areas of multilevel modelling, non-linear modelling and linear modelling to provide a background to the techniques currently used in inter-awarding body comparability studies. Issues surrounding continuing good practice in, and improvements to, the statistical comparison of grading standards are discussed.